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Preface 


For the thirty-ninth time, the Research and Theory Division of the Association for Educational 
Communications and Technology (AECT) is sponsoring the publication of these Proceedings. Papers 
published in this volume were presented at the annual AECT Convention in Jacksonville. FL. A limited 
quantity of these Proceedings were printed and sold in both hardcopy and electronic versions. Volumes | 
and 2 are available through the Educational Resources Clearinghouse (ERIC) System. Proceedings volumes 
are available to members at AECT.ORG. Proceedings copies are also available at: 


http://www.tresystems.com/proceedings/ 


The Proceedings of AECT’s Convention are published in two volumes. Volume #1 contains papers dealing 
primarily with research and development topics. Papers dealing with the practice of instructional 
technology including instruction and training issues are contained in Volume #2. This year, both volumes 
are included in one document. 


REFEREEING PROCESS: Papers selected for presentation at the AECT Convention and included in these 
Proceedings were subjected to a reviewing process. All references to authorship were removed from 
proposals before they were submitted to referees for review. Approximately sixty percent of the 
manuscripts submitted for consideration were selected for presentation at the convention and for 
publication in these Proceedings. The papers contained in this document represent some of the most current 
thinking in educational communications and technology. 
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Technology in Support of Diverse Assessment 


Gary L. Ackerman 
Windsor (Vermont) School 


Abstract 


This paper presents the case of a small rural school that implemented an assessment system designed 
around three different measures, each designed to answer a different question about the secondary students enrolled 
in the school. Technological solutions were used to captured both qualitative and quantitative evidence of students’ 
learning; those solutions are described and other findings are discussed. 

Keywords: assessment, K-12, portfolios, educational technology 


Technology in Support of Diverse Assessment 


Compared to previous generations, today’s educators gather more and more types of information regarding 
students’ learning and they use it to make more decisions (Rajagopalan & Gordon, 2016). Data gathered during 
instruction are used formative purposes and data gathered in summative assessments are used for evaluation (Harlen 
& James, 1997). Both types of assessments are used to improve curriculum, to evaluate programs, to justify 
accreditation decisions, and for other institutional purposes (Walvoord, 2010). Despite the long tradition of relying 
on the results of tests to measure students’ learning, there is increasing recognition that diverse assessment data are 
necessary to completely describe students’ learning, understand the value of instruction and curriculum, and identify 
aspects of schooling that need to be improved (Andrews & Wulfeck, 2014; Dede, 2010). 

As data and its purposes have become more sophisticated, information and computer technology (ICT) has 
been applied to many assessment problems, and a range of ICT tools have been deployed to facilitate assessment. 
Nearing the third decade of the 21st century, educational practitioners use ICT to collect easily quantifiable data 
such as scores on tests that are administered given to large populations (Bennett, 2002) and content questions that 
can graded by computers in online and blended classes (Black, Beck, Dawson, Jinks, & DiPietro, 2007). ICT has 
also been used to document and assess learning in classrooms where teachers seek to promote and document 
complex learning (Webb & Gibson, 2015). Further, ICT tools are used to manage vast collections of assessment data 
as well as to manipulate, analyze, and report those data. 

In this paper, the author presents the case of a public school in which ICT was adopted and adapted to 
facilitate several assessment practices. There are two purposes of this paper. First, to describe the assessment system 
and the roles of technology in the system. Second, to articulate and support conclusions regarding the design of 
effective technology systems in secondary schools. 


The Setting and the Data Collection 


The project described in this paper occurred in a comprehensive public school in a rural area of the 
northeast United States; it enrolls approximately 300 students in grades 7-12. The guidance counselor at the school 
described it as “similar to other schools in the state. We have slightly more students in poverty and receiving special 
education services than others, but we are closer to the middle than the extreme,” and the school “sends its share of 
students to Ivy League schools, but we also have many students who go to work right out of school.” 

In the summer 2015, several groups of educators in the school were working on different initiatives; those 
groups included a committee working to increase the use of project-based learning (Krajcik & Shin, 2014) in all 
courses; a dyad preparing reports for an upcoming accreditation review; and a group of four working to upgrade the 
information technology hardware, software, and web services for instruction. As the principal was preparing for the 
beginning of the school year, he convened a meeting of individuals representing those groups for the purpose of 
“designing assessments that bring together the curriculum, technology, and accreditation work so it all fits together.” 
In introducing the project to the group, the principal noted, “we have our students’ SBAC, PSAT, SAT, and other 
test scores, but our students do so much more. By capturing a more complete view of our students, we can improve 
our curriculum and reporting to the accreditors.” The members of this group agreed to assume a leadership role in 
defining a new assessment system. For the purposes of this paper, this group is called the Consolidated Committee. 
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The author was involved as an internal consultant to the Consolidated Committee. He was primarily involved in 
designing and configuring the technology solutions, and revising them as they were deployed. He was present at 
most meetings of the Consolidated Committee to provide input specific to the technology infrastructure and to 
understand what actions the committee recommended so that the technology accurately reflected the intent. 

This case is described with data from three sources. First, the agendas used to plan and the minutes used to 
record the meetings of the Consolidated Committee were reviewed. The primary purpose of this data was to 
accurately describe the work including design decisions and the rationale that supported each decision. Second, the 
author conducted a focus group interview (Fey & Fontanta, 1991) of the Consolidated Committee. Finally, the 
author conducted a second focus group interview with a group of teachers who had contributed to and made use of 
the assessment system. Both focus group interviews were conducted about 18 months after the project began and at 
a time when the systems were fully functional and active development had stopped (although the Consolidate 
Committee reconvened soon after the interviews were completed to continue development of the system). 

The transcripts of the focus group interviews were coded using the constant comparative method (Glaser, 
1965). The author along with a second researcher read and coded the transcripts of the interviews; themes that were 
identified by both researchers are elucidated in the findings. 


The Assessment System 


The Consolidated Committee adopted a planning method grounded in educational design research 
(McKenny & Reeves, 2012); this method is intended to facilitate the use of theory to create interventions that 
improve educational practice, and it comprises three phases (see Figure 1). McKenny and Reeves (2014) described 
educational design research as “the iterative development of solutions... to practical and complex educational 
problems” which leads to “new knowledge that can inform the work of others” (p. 133). Key aspects of educational 
design research include an iterative approach to understanding the local instantiation of a problem and continuous 
planning actions that reconcile theory with the local circumstances. Further, educational design researchers seek to 


both evaluate the degree to which the interventions produced the desired results and they seek to articulate and 
support generalizations that can inform similar efforts in different settings. 

The principal described the vision for the assessment system that emerged during conversations with the 
faculty over the course of several months prior to the formation of the Consolidated Committee. These became the 
analysis/ exploration phase of the process. The principal observed, 

We seem to agree that a rich and varied curriculum leads to good test scores, but we know those scores do 

not show the true extent of our students’ work. Plus, our local learning expectations [upon which] we spend 

so much time on don’t show up in any tests. Everything students do is important to us, so we need to 
capture the data in whatever way [the students] exhibit their learning. 


Analysis Design ee Reflection 
“What does the eae tell us?" What Systems, adaress similar “What can we generalize?" 
problems?" : 
"How is ic le antes POnstucHon Evaluation 
P "What will our community do?" “Did it meet our needs?" 


in our community?" 


Figure 1. Educational design research adapted from McKenny & Reeves (2012) 


In this vision, the principal and his faculty sought to broaden their definition of assessment data to more 
closely align with high-quality and valid assessments (Pellegrino, 2014) than is possible relying on standard test 
scores, which “had been the focus of assessment efforts previously.” The Consolidated Committee recognized the 
need for diverse data that documents creativity, problem-solving, communication, and collaboration to be included 
in the formal assessment system, as those skills have taken on new importance in organizations and cultures in 
which digital electronic information technology dominates (Black & William, 1998; Cumming & Maxwell, 1999; 
Cumming, Maxwell, & Wyatt-Smith, 2016; van Laar, van Deursen, van Dijk, & de Haan, 2017). 
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Dimensions of Learning 


Three sources of data became the focus of the assessment system (see figure 2), and IT systems were 
designed to collect and use all three types of data. While the Consolidated Committee used the terms “Course 
Grades,” “Standard Tests,” and “Performances” to identify the types of information they were collecting, they 
suggested the assessment system was designed to answer three questions: a) Does the student have the habits of 
effective learners and workers? b) How does each student compare to students in other populations? c) Can the 
student create polished work that is valued outside the context of school? and the three sources of evidence in the 
assessment were interpreted as answering those questions. 


Course Grades 


Standard 
Tests 


Figure 2. The three sources of evidence in the assessment system 


The students in the school participate in a range of standards-based and standardized tests. These include 
tests associated with college admissions, as well as the tests associated with the Common Core State Standards 
(CCSS) (National Governors Association Center for Best Practices & Council of Chief State School Officers, 2010). 
Because he led a public school, the principal was obligated to ensure these test scores were shared with the public, 
so it was decided the school would continue its participation in the tests. Students would be encouraged to fully 
engage in the tests and the scores would be presented to students, parents, and the community as “one small part of 
what we know students can do, and the only chance to see how they compare to students in the wider population.” 
Following this rationale, students’ scores on these tests became part of the assessment system. 

While many schools appear to be abandoning traditional course grades in favor of reporting on students’ 
progress towards standards (Stump & Silvernail, 2014), the Consolidated Committee reasoned learning in that 
dimension was already being documented in the Smarter Balanced Assessment Consortium (SBAC) tests 
administered to comply with CCSS, thus a standards-based course grade was redundant. Traditional course grades, 
on the other hand, are positively associated with several aspects of each student’s skills and knowledge; including 
content knowledge, ability to complete tasks in a timely manner, compliance, and attitude and participation which 
appear to be necessary for many workplace settings (Manpower Report, 2015). The committee recorded agreement 
with the teacher who suggested, “these are all things that matter when students leave the school, no matter where 
they go, shouldn’t we somehow record their ability to succeed in similar ways in school?” They further reasoned the 
subjective assessment of students by different professionals in different courses was a predictor of students’ success, 
thus they decided to continue collecting traditional course grades. 

The final dimension of learning that needed to be captured in the assessment system was students’ 
participation in project-based learning (Krajcik & Shin, 2014) and other authentic learning environments 
(Herrington, Reeves, & Oliver, 2006) in which the products they created resembled the work done by professionals 
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working in their field of study. The Consolidated Committee recommended an electronic portfolio platform with 
which students could collect and cull artifacts (mostly images and video) of the projects they created along with 
their reflections on the importance of that work be included to complete the assessment system 


Technologies for Assessment 


Once the three dimensions of learning had been defined, the Consolidated Committee turned its attention to 
designing ICT systems that would allow teachers and others to easily record data documenting learning in each 
dimension and that would allow school leaders, teachers, and others to manipulate and present the data that 
documented each dimension of learning. Specifically, the committee decided to adopt three systems. 


In the summer when the Consolidated Committee began designing the new assessment system, the student 
information system (SIS) was upgraded as part of a long-established technology plan. The Consolidated 
Committee also recognized the school had been using the SIS for several years and that “teachers, parents, and 
the administration along with the technology staff were familiar with and satisfied with the method of recording 
course grades,” so that system would remain in place and be used to document course grades. 

A spreadsheet file would be created to track students’ scores on standards-based and standardized tests that are 
administered to high school populations throughout the state, region, and country; this was known as the 
“Dashboard Spreadsheet” following the convention of displaying quantitative assessment data on dashboards. 
A new electronic portfolio system to capture artifacts of students’ performances and products on authentic 
projects was to be installed and configured. 


The first design of the Dashboard Spreadsheet compiled students’ SBAC, SAT, PSAT scores. Upon 
reviewing that draft, the Consolidated Committee concluded there was too little data, so it was immediately 
redesigned; this is consistent with the design/ construction phase of educational design research (McKenny & 
Reeves, 2012). Changes included adding grade point averages and categorical data such as foreign language courses 
passed, participation in athletics, arts, and other activities. Included in this first design/ construction iteration was 
defining procedures for exporting relevant data from the SIS and importing it into the Dashboard Spreadsheet. 
Further, formulas were added to the spreadsheet to automate the calculation of descriptive statistics. 

Once that second design was completed, the Consolidate Committee reviewed it and found it more 
complete and satisfactory given the anticipated uses of the data to recommend student choices for courses, respond 
to leaders’ requests for information, and otherwise improve the school curriculum. The Consolidate Committee also 
sought input from the professionals who managed the SIS to confirm the work of exporting data from the SIS did 
not significantly increase that person’s duties. One additional design iteration was undertaken so that correlations 
between data sets contained in the Dashboard Spreadsheet could be displayed graphically and numerically. 

At the end of the initial design/ construction session and the two following iterations, the Consolidated 
Committee reported, “the dashboard is ready for use, and it should be populated with real data.” A final design/ 
construction iteration followed during which two individuals from the Consolidated Committee, along with the 
author met to improve the appearance of the data, lock cells on the spreadsheet to prevent accidental editing of data, 
anonymize data as appropriate to maintain privacy, and otherwise finalize the file. 

The second new ICT tool designed for the assessment system was an installation of Mahara (n.d.), an open 
source platform for creating, curating, and presenting electronic portfolios. In adopting this platform, the 
Consolidated Committee sought to resolve the problem of attempting to use the SIS to capture authentic student 
work. They observed, “not only are the comments we have been using difficult to manage, they don’t seem well- 
understood by parents, and it is impossible to record students’ reflections.” This system also underwent several 
design/ construction iterations. 

The first design established “’ proof-of-concept’ as no one on the committee has any experience using 
Mahara, so we want to see it and determine of it is an improvement over [the previous platform].” Two additional 
design/ construction iterations followed. During the first, Mahara was reconfigured to be integrated with the learning 
management system that had been adopted by the school. The LMS had been adopted under an earlier initiative to 
deploy a one-to-one computing initiative, but it had not been widely used. Once two LMS and Mahara were 
integrated, the second design iteration added rubrics to the LMS and configured the systems so Mahara portfolios 
could be submitted to the LMS. The Consolidated Committee recommended, “all assessment and reporting of 
projects and authentic learning be migrated to the LMS and Mahara. This will allow for full documentation of 
learning and reporting of qualitative data exclusively through this system.” 
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With the three technology-based assessment tools, the system was completed. The SIS was used to display 
traditional course grades, and also as a source of data for the Dashboard Spreadsheet. After several drafts, the 
dashboard displayed both individual student data and data summarizing entire cohorts of students. Finally, the 
electronic portfolio platform was integrated with the LMS and configured to capture ad display artifacts of works 
created by students and associated qualitative data. 


Designs of Technology-Rich Assessment 


The final phase of educational design research (McKenny & Reeves, 2012) is evaluation/ reflection during 
which the researcher-practitioner both determines the degree to which the interventions solved the problem as it was 
locally instantiated and articulates generalizations that can be used to solve similar problems in different settings. In 
this case, there appears to be support for three observations regarding the design of technology-rich systems to 
support assessment in schools; the system is evaluated in the Discussion. 


Efficiency. 


One advantage of using ICT to manage the vast assessment data is the efficiency with which sophisticated 
analyses can be deployed. When initially designing the Dashboard Spreadsheet, the Consolidated Committee sought 
to minimize the amount of data that would need to be input. During design/ construction, they discovered the data 
was received in the school or existed in the SIS in formats that could be imported with minimal effort. Batch 
uploading and simply copying and pasting allowed “us to enter data far more quickly than expected.” The 
Consolidated Committee reported, “in the time we apologized for asking [the SIS manager] to get us the data we 
wanted, he had it into the dashboard.” Further, the formulas used to generate the descriptive statistics could be 
written into a single spreadsheet file which was saved as a template, so creating the dashboard for new students and 
for new cohorts was “the matter of unlocking the file and making a few clicks.” 

Because Mahara was a new tool for all faculty and because the faculty expressed some reluctance to adopt 
it because previous platforms had proven ineffective and were abandoned, the Consolidate Committee deployed the 
system in steps. For example, as faculty gained experience with Mahara, they used paper copies of rubrics to assess 
students’ work. As they become more familiar with the platform and realized digital rubrics in the LMS could be 
used to reduce the time needed to provide and review feedback, the faculty asked for the design/ construction 
iteration during which rubrics were added to the integrated LMS and Mahara system. 


Ease of Use. 


For several years, the school had been attempting to increase the use of online classrooms as well as 
consistently develop electronic portfolios; those efforts had largely been unsuccessful because “the systems all 
required a separate log on.” With the single step of allowing single sign-on so that the LMS and Mahara used the 
same credentials as students and teachers used to access email and other web-based services, this assessment system 
was found to be easier to use. Teachers reported, “it seemed a simple barrier, but having even one or two students 
fighting to remember how to log on made us avoid them. Now that the systems are integrated, my department uses 
both [LMS and portfolios] more than we did.” 

Ease of use was also identified as an obstacle to using the dashboard to the extent it was intended. 
Specifically, teachers found it difficult to customize the graphs so they displayed the correlations between scores and 
other data in which they were interested. While most agreed the steps were not difficult, they were unfamiliar, so 
many avoided the need to learn by finding a colleague who was willing to create the graphs he or she needed. 


Understanding Learning. 


The final observation that was consistently reported by both the Consolidated Committee and the faculty 
who used the assessment system was that it had increased their understanding of the degree to which students were 
learning. Their understanding appeared to be for both individuals and for cohorts. 

The addition of qualitative data regarding students’ projects and authentic work in the electronic portfolios 
“gave [teachers] insight into how students were applying what we studied in class.” Teachers also reported the 
technology was one factor that contributed to their ability to gain this insight. “Because the system was always 
available and easy to use, it was used by more teachers than the other portfolios and they added more work so we 
had more to look at,” observed on teacher, and that was followed almost immediately be another who added, 
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“having the rubrics there also made it more clear to us what we thought we should be seeing, so I could focus on 
what the [student] did and what we thought they should be doing.” The qualitative data that was consistently gathers 
and easily assessed led to deeper understanding of individual students. 

The Dashboard was found to be particularly useful for understanding students as a cohort. The fact that 
data was compiled for all of the students in a grade in a single place made data available in a manner that it was not 
previously. “Especially for the students in the middle quartiles, the dashboard seemed to given of information we did 
not have previously,” reported the teachers. “We could see differences between what they had taken for courses, for 
example, and see that those who had taken certain classes appeared to have higher tests scores. This is useful for 
making recommendations,” added the principal. 


Discussion 


There two purposes of this paper. First, to describe the design of the assessment system. In this case, the 
technology-based assessment system appeared to provide educators with information that they did not have prior to 
the deployment of the system. The additional data appeared to be both the result of the system being easier to use 
than other systems and the system being designed to gather data that was previously lost or not collected. Because 
the system was designed through iterative processes with changes being recommended by users of the system, it 
appears to have more closely reflected their needs and their capacity to use it. 

Second, to articulate generalizations regarding the design of ICT systems in school. These data appear to 
support the conclusion that the unified theory of acceptance and use of technology (UTAUT) (Venkatesh, Morris, 
Davis, & Davis, 2003) accurately describes the factors that predict the use of ICT in educational populations. 
Specifically, UTAUT posits effort expectancy is positively associated with the intention to use technology. In this 
case; case of use and efficiency, both factors associated with effort expectancy, were reasons teachers cited for using 
the assessment system. Also, performance expectancy, which is grounded in job fit is posited to positively affect 
technology use. In this case, teachers claimed the system helped them understand their students learning, thus they 
were abler to perform essential job function. 

While this project does appear to have resulted in an assessment system judged to be an improvement over 
those used previously, and this improvement is recognized as an aspect of user-based research (Bereiter, 2002), it is 
unknown the degree to which this improvement was recognized by all stakeholders. Because the assessment system 
was an initiative begun by the principal who had supervisory authority over all of the faculty, and the faculty is 
fewer than 30, it was difficult to identify potentially dissenting voices. Further, it is unknown the degree to which 
this specific effort and the iterative nature of it was responsible for the observations. Given the authority of the 
principal, any new assessment system could have produced these results. 


Conclusion 


Despite the limitations for the research, there does appear to be support in these data that an assessment 
system that used ICT configured to provide both quantitative and qualitative data regarding students’ performance in 
three dimensions of learning contributed to teachers understanding of their students. Both the data they already 
gathered were more easily manipulated and displayed and they were able together data the was previously lost 
because it was not consistently collected. 


References 


Andrews, D. H., & Wulfeck II, W. H. (2014). Performance assessment: Something old, something new. 
In Handbook of research on educational communications and technology (pp. 303-310). Springer New 
York. 

Bennett, R. E. (2002). Using electronic assessment to measure student performance: Online testing. State Education 
Standard, 3(3), 23-29. 

Black, E. W., Beck, D., Dawson, K., Jinks, S., & DiPietro, M. (2007). Considering implementation and use in the 
adoption of an LMS in online and blended learning environments. TechTrends, 51(2), 35-53. 

Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: principles, policy & 
practice, 5(1), 7-74. 

Bereiter, C. (2002). Education and mind in the knowledge age. Mahwah, N.J: L. Erlbaum Associates. 

Cumming, J., & Maxwell, G. (1999). Contextualising authentic assessment. Assessment in education: Principles, 
policy & practice, 6(2), 177-194. 


154 


Cumming, J., Maxwell, S., & Wyatt-Smith, C. (2016). School leadership in assessment in an environment of 
external accountability: Developing an assessment for learning culture. In Leadership in Diverse Learning 
Contexts (pp. 221-237). Springer International Publishing. 

Dede, C. (2010). Comparing frameworks for 21st century skills. In J. A. Bellanca (ed.) 2/st century skills: 
Rethinking how students learn (pp. 51-76). Bloomington, IN: Solution Tree Press. 

Frey, J. H., & Fontana, A. (1991). The group interview in social research. The Social Science Journal, 28(2), 175- 
187. 

Glaser, B. G. (1965). The constant comparative method of qualitative analysis. Social Problems, 12(4), 436-445. 

Harlen, W., & James, M. (1997). Assessment and learning: differences and relationships between formative and 
summative assessment. Assessment in Education: Principles, Policy & Practice, 4(3), 365-379. 

Herrington, J., Reeves, T. C., & Oliver, R. (2006). Authentic tasks online: A synergy among learner, task, and 
technology. Distance Education, 27(2), 233-247. 

Krajcik, J., & Shin, N. (2014) Project-based learning. In R. K. Sawyer, (Ed.). The Cambridge handbook of the 
learning sciences (Second edition) (pp. 298-318). New York, NY: Cambridge University Press. 

Mahara [Computer Software]. (n.d.). Retrieved from https://mahara.org/, August 28, 2015. 

Manpower Group (2015). Talent shortage survey: 10th annual talent shortage survey. Retrieved on March 12, 2016 
from The Manpower Group Web site: http://www.manpowergroup.com/wps/wem/connect/408f7067-ba9c- 
4c98-b0ecdca74403a802/2015 Talent Shortage Survey- 
lo_res.pdf?MOD=AJPERES&ContentCache=NONE 

McKenney, S., & Reeves, T. (2012). Conducting educational design research. New York: Routledge. 

McKenney, S., & Reeves, T. (2014). Educational design research. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. 
Bishop (Eds.), Handbook of research on educational communications and technology (pp. 131-140). New 
York, NY: Springer New York. 

National Governors Association Center for Best Practices, & Council of Chief State School Officers. 

(2010). Common Core State Standards. Washington DC: Author. 

Pelligrino, J. (2014). A learning science perspective on the design and use of assessment in education. In R. K. 
Sawyer, (Ed.). The Cambridge handbook of the learning sciences (Second edition) (pp. 233-252). New 
York, NY: Cambridge University Press. 

Rajagopalan, K., & Gordon, E. W. (2016). The testing and learning revolution: the future of assessment in 
education. Springer, Chicago. 

Stump, E., & Silvernail, D. (2014). Implementation of a proficiency-based diploma system: Early experiences in 
Maine. Gorham, Maine: Maine Education Policy Research Institute in the School of Education and Human 
Development, University of Southern Maine. Retrieved from http://digitalcommons.usm.maine.edu/ 
cepare_proficiency/2 

van Laar, E., van Deursen, A. J. A. M., van Dijk, J. A. G. M., & de Haan, J. (2017). The relation between 21st- 
century skills and digital skills: A systematic literature review. Computers in Human Behavior, 72, 577— 
588. 

Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User Acceptance of Information Technology: 
Toward a Unified View. MIS Quarterly, 27(3), 425-478. 

Walvoord, B. E. (2010). Assessment clear and simple: A practical guide for institutions, departments, and general 
education. John Wiley & Sons. 

Webb, M., & Gibson, D. (2015). Technology enhanced assessment in complex collaborative settings. Education and 
Information Technologies, 20(4), 675-695. 


155 


Changing with the Times: How Do We Lead Technology Integration, 
Including Mobile Devices, in Schools? 


Stacie Barnett-Slusher 
Morehead State University 
P.O. Box 130 
Calvin, KY 40813 


Abstract 


Because of the availability and growing use of mobile technologies and communication models, 
“Enthusiasts argue that the presence of such technologies would push schools in the direction of embracing the 
liberating possibilities of new media rather than limiting their use through acceptable use policies” (Collins & 
Halverson, 2009, p. 26). Unfortunately, some schools are not enthusiastic about the presence of mobile technologies 
and restrict use through policies. Regarding such policies, schools vary widely in their approach and acceptance of 
integrating these technologies, especially mobile devices. Some schools limit access and exercise control of the use 
mobile devices through restrictive or outdated policies and procedures governing both employees and students, 
while other schools have more welcoming “bring your own device” (BYOD) policies. 


Introduction 


“For it is inescapable that every culture must negotiate with technology, whether it does so intelligently or 
not” (Postman, 1992, p. 5). Historically, new technologies are viewed with wonder and/or suspicion, especially in 
educational institutions. Some challenges to technology integration are due to caution, change resistance, existing 
infrastructure, and resistance to disruption. Some technologies prove unable to meet the needs of the customer 
(institutions, systems, teachers, and students) better than traditional methods or established technologies; therefore, it 
is naturally difficult and unlikely for those technologies to form a disruption. On the other hand, the prevalence of 
technologies, such as mobile devices, is responsible for new customer demands. 

Previously, many technologies available within the school environment required an institutional purchase 
and were not readily available to the end-users because the school determined both if and when the technology 
would be available. The prevalence of mobile devices that are owned and supplied by the user changes this dynamic. 
The shift of availability and control has changed. “Hence the imperative of technology is toward more learner 
control, and schools are fighting a losing battle to control what students learn” (Collins & Halverson, 2009, p. 18). 
The result means that customers (students) and members of educational institutions have the potential to drive 
integration, if not disruption. 

Similar to the negative stance the American Federation of Teachers once took regarding video use, some 
institutions feel that the opportunity for misuse is too great. The National Education Association (NEA) and many 
other institutions approach device use as something they can control, asking students and employees to only access 
the Internet via school networks that are governed by such policies. Since many users not only have their own 
device, but have independent access to the Internet through personal data plans, is it realistic to expect that schools 
can exercise complete control over these technologies? Are schools refusing to change with a technological change 
that they don’t own? 

Decreasing the digital divide, or the access to technology, has long been a concern. More recently, the 
concern has evolved to a second-level digital divide, the ability to use technology. Literature in the early 2000’s 
through the present reflects the second-level digital divide, “the difference, or “divide,” in how technology is used” 
(Reinhart, Thomas, & Toriskie, 2011). Wei, Teo, Tan, & Chan (2011) refer to the second-level digital divide as a 
“digital capability divide.” The confounding issue regarding mobile technologies is that access is available, but 
institutions are limiting students’ use that can increase proficiency beyond purely social applications. 

Mobile technologies provide an opportunity for students to learn and use technology skills for applications 
are common in the workplace and outside of a school environment. Furthermore, using available technologies 
increases students’ 21‘ Century Learner skills. The Framework for 21‘ Century Learning is a set of standards meant 
to prepare students “to thrive in today’s digitally and globally interconnected world” (Partnership for 21st Century 


156 


Learning, n.d.). These standards are intended to prepare students to respond to and be adaptable to rapidly changing 
technologies. In addition, it is equally important to realize that today’s students will use technology in their everyday 
lives. Access and proficiency in technology is directly related to access to information, education, healthcare, and 
employment opportunities. 

A discussion within a community of practice addressed some historical experiences with “new” 
technologies in schools, such as the belief that radio, then television, would revolutionize education and seek to 
compare past technology changes to modern technologies. Based on experiences in learning communities and 
organizations, policy reviews, and surveys; existing policies were presented, and the educational technology 
practitioners shared their own experiences with policies and procedures in a variety of educational settings as both 
professionals and as students. 


Participants engaged in the following discussion points: 


Do schools spend more time fighting the technology instead of embracing it? 

How much control is feasible? 

What are the policies and procedures that allow the best chance of technology integration? 

Should schools look at the technologies as a possible disruption if the customers can meet their needs and 
still use a device that they aren’t willing to allow the schools to govern? 

Does the wide range of policies add to the digital divide? 

What is the role of educational technology professionals in leading change? 


Conclusion 


Schools are traditionally change resistant and slow to embrace change. Generally, change and “successful 
implementation depends on the commitment of top management” (Januszewski & Molenda, 2008, p. 183). As 
professionals and educators, it is time to recognize that these technologies do not fit the mold of some current 
policies or the needs of students. Schools no longer can ensure control by not purchasing, not using, and 
condemning new technologies. The users have some control and the playing field has changed. Schools will not be 
able to control all tablets, phones, watches, and other mobile technologies. Perhaps it is time to lead the change to 
implementation and embrace the learning potential for users rather than focusing on restriction. If “the end purpose 
of educational technology is using” as suggested by Januszewski and Molenda (2008, p. 168), then leaders should 
encourage institutions to transform policies and views regarding technology and mobile devices into ones which 
demonstrate appropriate use and usefulness in learning and communication. 
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Abstract 


STEM Education is a critical component of high quality 21st-century education, but increasing demands 
are placed on teachers to prepare students for state and federal exams. This paper will detail three model public 
school districts that are currently using individualized digital tutors as a tool to reduce the classroom time that test 
preparation takes, allowing teachers to engage their students in real-world, problem-based STEM learning activities. 


The Importance of STEM Learning 


Science Technology Engineering and Mathematics, or STEM, along with its many variations (STEAM, 
eSTEM, STEMM, etc), represents a movement in the United States to expose students to 21st century learning 
environments (White House, 2016). In our evolving technological world, students need to acquire STEM skills to be 
college and workforce ready, even if they are not going to be an engineer or scientist. These resources come in 
different forms, such as programmable robots, sandbox style video games, building tools, and virtual and augmented 
reality devices. These learning materials give students a chance to problem solve and to be creative in ways that 
traditional textbooks and testing can not measure. Unfortunately, there currently exists a tension between preparing 
for high-stakes end-of-year testing, and allowing time for exploration of relevant 21st century skills for students. 
This paper outlines the aggregate experiences of three public schools that have managed, with only a small amount 
of outside help and a lot of creativity, to square this circle, ensuring high performance on end-of-year tests, as well 
as allowing the freedom for teachers to engage their students in problem-based STEM learning. 


Common Core and State Testing 


While there are currently over 20 schools participating in the NEF STEM+ Academy program, the three 
schools we are focussing on are Canton Central School District in Canton, NY; Steubenville City Schools in 
Steubenville, OH; and Martins Ferry City School District in Martins Ferry, OH. All three are small-to-moderate 
sized school districts with relatively limited resources. Schools around the country continue to focus on student 
testing, and recently on teacher evaluations (Educational Leadership, 2016). Finding their teacher evaluations tied to 
student performance, the teachers in these districts found themselves in an untenable situation: they felt that while 
previously there may have been time to explore new and relevant topics not directed by the state testing 
requirements, preparing students now took up virtually all of their classroom time. Intentional or not, the cultural 
shift to double-down on high-stakes testing and teacher evaluations has in many ways hurt the public school system, 
in many cases reducing teaching to a lowest common denominator: “The Test.” 
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People and Organizations Willing to Help 


Rather than giving in to the status quo, the teachers at these three schools sought outside help and received 
funding from local businesses, nonprofits, and other sources to provide individualized digital tutors and STEM 
learning for students at different grade levels in each of the districts (Batt, 2016). A local private foundation 
provided partial funding for Canton, while Steubenville and Martins Ferry used funds from their own district 
budgets. The remainder of funding came from a mix of grants, donations, and nonprofit partnerships. The graduate 
program in Educational Technology at The State University of New York at Potsdam partnered with one of these 
nonprofits, and worked with these districts to provide professional development and logistical support in managing 
the implementation of the individualized learning system. The program the teachers from these districts have 
implemented represents some of the best ideas from the sixteen STEM+ academies in ten states around the country 
(Canning, 2016). 


A “Systems Solution” 


Our approach acknowledges and works with the idea of a school district as an integrated system of 
students, teachers, parents, administration, and community members, and provides tools for all of these groups: the 
parents and community are provided with professional development software, allowing them to build their skills in 
IT, Business, and Desktop Applications; the administration is provided with additional funding and teacher 
development; the teachers are given access to 21st century learning tools; and the students are provided personalized 
learning software that is customized to their own particular strengths and weaknesses. Students are also provided 
with motivational incentives, including Olympic Style Leaderboards, individual rewards, and group-based rewards, 
all based on their performance. Because performance is measured by their gain over their initial benchmark, students 
of all abilities are competing on a relatively level playing field. 


Conclusions 


In the three years since we have begun working with these districts we have seen a strong positive 
correlation between the students’ performance in the individualized learning software and their state test scores in 
both Math and Reading. For instance, at Canton, 100% of the students who were at or above grade level in Math and 
Reading in the learning system at the time of their state test passed that test. Similar results were found at the other 
two schools as well. When armed with some clarity and assurance of their students’ state test scores, the teachers felt 
less obligated to “teach to the test” and started introducing more and more STEM learning activities in the newly 
created free time, such as K’Nex building tools, Snap Circuits, Minecraft software, and programmable robots into 
their classrooms.The students, as might be expected, were thrilled, so much so that many asked their parents for 
these tools and kits as Christmas gifts. 

The system quickly became self-reinforcing, as access to STEM learning was predicated on student 
performance in the individualized digital tutoring system. As students succeeded in the tutoring system, they gained 
access to STEM learning activities; the more they engaged in those activities, the more they wanted to continue with 
them, which spurred them to continue to strive to do well in the tutoring system to secure this access. Because the 
individualized tutoring software is scaffolded to state standards, their performance on the state tests increased. 
Regardless of which is more important - the state test scores of the engaged student learners - what was previously 
seen as an issue of either-or was transformed into a yes-and. 

Because the systems at these schools communicate with each other through their University and nonprofit 
partners, the change seen in these schools is spreading to other school districts and states. New partner schools are 
already learning and replicating their success, and while the fundamentals of the system are the same, each school 
has a unique way of putting the pieces together. Changing the systems in three local systems is but a start: a proof- 
of-concept showing that performance on state tests and 21st century STEM learning can coexist and complement 
each other. The next step is to share these results with other stakeholders, and to continue on the mission that 
President Barack Obama set forth for universities and schools, that is to make students “...more excited than ever 
about pursuing their passions for STEM.” (Ransom, 2016). 
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Abstract 


Grounded upon the Diffusion of Innovations theory (Rogers, 2010), a prototype of a game for social 
improvement was developed to begin the process of systemic change. The purpose of the game is to disseminate 
knowledge, the first step in a systematic process to affect changing mindsets. The content’s focus is on the United 
Nations (U.N.) Convention on the Rights of the Child (CRC) Treaty of 1989, precisely Article 12: The Right to be 
Heard and Article 13: Freedom of Expression (The U.N. International Children’s Emergency Fund [UNICEF], 
2009). The Treaty was created to be a beacon of light for all children. Quite simply, it is a compilation of basic 
human rights children should be afforded. All U.N. nations have ratified the CRC Treaty but one. Despite American 
presidents taking an active role in initiating and supporting this call to action, the United States still abstains from 
ratifying it since its inception in 1989 (UNICEF, 2009). An endogenous game was selected as the vehicle to 
immerse the players in real-world scenarios with additional elements of fantasy to motivate engagement (Gunter et 
al., 2008). Through simulated experiences, the game was designed to provide education about the given topic while 
the pathos was intended to evoke emotion and change attitudes (Gee, 2008; Jones, 2008). It was hypothesized that 
this knowledge and change in attitudes could potentially induce action leading to real-world improvement. 
“Knowledge is power” (Bacon, 2001) and the first step to begin change. 


Introduction 


Clearly the U.S. provides many protective services for child welfare and, in fact, has a constitutional 
amendment which affords freedom of speech for all its citizens. Is there a need for instruction to teach about rights 
for children with so many provisions already in place? For the children who directly encounter the inequitable 
unwritten practices in our constitution and judicial system, these provisions are not enough to protect their human 
rights. Akhil Reed Amar proposes: 

The eight thousand words of America’s written Constitution only begins to map out the basic ground rules 

that actually govern our land, while the unwritten perceptions guide all other judgment (2012, p.ix). 
Historically, the practice has been that adults know what is best for children and make those determinations 
regardless of conflicting opinions from the children (Rodham, 1973; Taylor, 2009). Examine the disparity within 
the court system. Criminal cases have acknowledged that children are capable of expressing their views such as in 
the Gault Case of 1967 (Federal Judiciary, 2015). The Juvenile Detention division may deem children “fit to be 
heard” yet these same children may be considered “unfit” in the Family Law division (Taylor, 2009). Likewise, 
children’s rights vary by state and county. Forty states provide children a voice in their own dependency cases 
forming an inequality for the children in the other states (Breger, 2010; Rodham, 1973). These rulings do not stem 
from research reflecting the children’s ability to express themselves better in one court of law over another but from 
legislative traditions. 

What is the significance of raising awareness about a treaty of human rights designed to protect these 
citizens who go under the radar? Studies reflect that over half of the marriages in the U.S. end in divorce (Centers 
for Disease Control and Prevention, 2014). If just half of those marriages have the average number of two children 
per household, over a million children may be in legal proceedings affecting them. Unlike the rights of the adults, 
however, these children’s fates are decided for them as they are often divided like the assets in the household. The 
decision of whether their words are considered is left up to a judicial system often not trained in mental health issues 
needed to make these determinations (Lemon, 1999). Even when the children are heard, their words are often 
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discounted based upon engrained practices (Collins, 2012). The numbers of unheard voices rise considering the 
almost two million children born out of wedlock and the divorced couples with multiple children (Centers for 
Disease Control and Prevention, 2014). Add in the element of domestic violence (DV) which in many cases is not 
even considered relevant in custody cases and the need for change becomes alarming. In fact, data reflects that more 
domestic violence victims die at the hand of someone they were in a relationship with than someone unknown to 
them (Haddix, 1996). A study of survivors in DV shelters reflects that majority of their children witnessed the abuse 
(Haddix, 1996). Evidence reflects that children are, in fact, an integral part of these cases and have the capacity to 
express their views (Holt, Buckley, & Whelan, 2008). These views are often not even taken into consideration in the 
extreme cases. The children in cases: James M. of California and Bartasavich v. Mitchell in Pennsylvania witnessed 
one of their parents kill the other parent. The assailants still had custody rights to the children upon their release of 
incarceration regardless of the children’s words (Haddix, 1996). 

This game-for-social-change was designed to educate and advocate for the equal rights of all children. 
Overall, the CRC Treaty contains fifty-four articles and four themes: Survival, Development, Protection and 
Participation. The latter was the focus of the initial instruction for this first prototype, specifically Article Twelve: 
The Right to be Heard, and Article Thirteen: The Freedom of Expression. The ultimate goal of the instruction was 
designed to teach digital natives about the need to support the ratification of the CRC Treaty in the U.S., with the 
ultimate goal of empowering the learners to take action. This digital native population was selected as they are most 
accustomed to learning with digital-game-based learning, systematic thinking and processing with simulations, 
(Prensky, 2001). 


The Background 


The idea for a game to teach originated from this researcher’s experience of witnessing game based 
learning motivate digital natives to play and learn core subject matter. Reasoning and critical thinking became by 
products of this problem-solving instruction (Gordon, 1970). Research supported the advancement of the idea. In 
fact, studies reflect Digital Game-Based Learning (DGBL) is a motivational means to learn (Justice & Ritzhaupt, 
2015), demonstrates improved focus, learning cognition (Eichenbaum, Bavelier, & Green, 2014) and retention 
(Brent & Felder, 2009; Gee, 2008; Gordon, 1970). Gee posits that brain function is based upon the embodiment of 
sensory involvement (2008). Video games, therefore, serve as a logical platform for learning based upon how the 
brain actually works. Gamers are provided with a problem and select avenues of advancement based upon the 
information they have. Through game stimuli players advance or retreat, learning through each phase of the 
experience (Gee, 2008). In fact, these are the same critical-thinking and logic skills required as a basis for Science, 
Technology, Engineering and Math (STEM) careers (Roberts, 2002). 

In a diverse and troubled world, gaming could provide an equal playing field for all students to learn while 
promoting choice and self-expression. Failed choices lead to learning the content rather than failing the lesson 
(Jones & Chang, 2014; Justice & Ritzhaupt, 2015). Gaming brings Bloom’s taxonomies to education in a fail-safe 
environment including: facts and information, comprehension opportunities, skills to master analysis and synthesis, 
leading into the cognitive domain where player-learners are challenged to review the process of how to excel from 
one level to the next (Moriarty, 1974). 

To assure sound pedagogy in meeting the goals of this instruction, the design principles stemmed from the 
First Principles of Instruction (Merrill, 2002a, 2002b, 2007, 2009) and focus on e3 Instruction: effective, efficient, 
and engaging (Merrill, 2007, 2009). Activation begins with an introduction of the CRC Treaty and the basic list of 
rights it provides to children. A CRC icon (envisioned to be a scroll in the game) is noted in the prototype to be 
accessible in each scene for the player to use as a resource to see all the articles as well as which countries have 
ratified the CRC. Players demonstrate understanding of content by selecting the accurate answer based upon 
whether the CRC Treaty was ratified for that country. According to selections made, the next path provided reflects 
the consequences of decisions. Branches of support resources and next game level of adaptive instruction empowers 
players to build self-efficacy and take action as they proceed. 

Aligning to the Elaboration Theory which suggests curricula should be built from the simple to the 
complex (Reigeluth, Merrill, Wilson, & Spiller, 1980), this instruction sequences and builds upon the foundational 
knowledge achieved from decisions made during each scenario. Thus, the scenarios advance from simple desires 
children may express to more important human rights issues. The tone is intended to be serious although the game 
begins with some humor to capture the attention of the player. Topics range from requests to be heard regarding 
breakfast choices to life-altering pleas to sleep in a safe home. Advanced layers include application activities for 
players to learn more about the topic and take action; travel through more scenes; and explore avenues intended to 
empower players with confidence building tips such as opportunities to select appropriate communication responses. 
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As the storyline unfolds, real-world scenes are presented in which player-learners make decisions which in turn 
teach them about the purpose of the CRC Treaty. Emerging branches support the player-learners in coping with 
feelings of sadness and empower them with feelings of hope as they learn about self-efficacy tips and actions they 
may take to survive the scenarios because affective domain can affect learning (Dormann, Whitson, & Neuvians, 
2013). 

Furthermore, DGBL was selected as the best viable option to disseminate knowledge, globalize and 
transforms education (Pellegrino & Scott, 2004) as DGBL is a platform players are already familiar with engaging 
in for hours at a time. Access is available to most users through mobile devices, computers and game consoles. Data 
reflects that the age range expands beyond simply the digital native and has generated in surplus of sixty billion 
dollars (Jones, 2008). The value of this DGBL, however, extends beyond dollars as its purpose focuses on changing 
attitudes, perceptions and potential social change for improvement. This game-for-social-change instruction elevates 
to the level of serious-games (Chen & Michael, 2006; Cody, Ritterfeld, & Vorderer, 2009; Heeter, Lee, & Peng, 
2010; Ratan & Ritterfeld, 2009). Downloaded over two million times, Darfur is Dying is one such example of the 
extent to how serious games can spread a message (Jones, 2008). Furthermore, gaming in education has far greater 
capabilities of influencing player-learners than traditional methods of instruction as evidenced in the article Can a 
Video Game Make Someone Nice? (Kapp, 2012). This study examined participants engaging in socially positive and 
negative games. The scenario ranged from docile activities like picking up spilled pencils to stressful situations such 
as stopping an abusive lover. In both cases, player-learners who played the game persuaded others to make positive 
choices and were more likely to come to the aid of another (Kapp, 2012). 

Gaming has also been used as a platform to educate about other treaties the U.N. has created. For 
example, the U.N. called for the decade from 2005 to 2015 to focus on education promoting sustainable 
energy. Answering the call, Ball State University in Indiana planned the development of the Second Chance 
Game: Local Partnerships (University-Community) for Global Awareness and Responsibility. The results of the 
DGBL initiative proved that imparting social knowledge and getting buy-in from key players to develop 
innovative strategies to educate goes beyond governments creating treaties or passing laws. (Pacheco, Motloch, & 
Vann, 2006).-Likewise, Europa games developed by the European Commission teach about the CRC to children 
in countries which had adopted the Treaty (EUROPA, 2011). These are exogenous games, based upon a 
behaviorism approach and reinforcement theory. 

Justice Sandra Day O’Connor spearheaded the use of games to teach about civics and US Government 
(Toppo, 2015). Since 2009, over three million users have played the free iCivics games to learn about the process of 
government and civics, such as how judges make rulings on landmark cases to protect rights (Toppo, 2015). Lastly, 
the European Union Commission, EUROPA, created web-based free games specifically designed to inform children 
in countries about their rights in countries which had ratified the CRC (Europa, 2011). 

Lastly, this prototype was modeled after games promoted by the Games for Change Organization 
(http://www.gamesforchange.org/), adding pathos through the real-world scenarios to deliberately evoke empathy 
and transfer of knowledge (Edele, Dziobek, & Keller, 2013; Gunter, Kenny, & Vick, 2008; Huang, & Tettegah, 
2010). 


The question is this: Can a game designed to educate about the U.N. CRC Treaty persuade the digital 
natives to ratify it in the U.S. in order to begin the movement for change to ultimately achieve equal rights for 
children? It is hypothesized that there is evidence to support that a game would be an effective means to advocate 
and profoundly impact invaluable social change to better the lives of children in need. It was time to create this 
game to teach the U.S. children about the CRC Treaty. 

“Never believe that a few caring people can't change the world... indeed, that's all who ever 

have,” Margaret Mead. 

The Game-for-Social-Change Prototype was the first step in this endeavor. Artifact URL: 
www.purpose2day.com/site/ChildrenAroundtheWorldGame-Prototype.html 


The Conclusion 


The researcher designed this prototype based upon a Choose Your Own Adventure architect (Packard, 
1979). Crowdsourcing returned real-world scenes which changed the persona of the characters and depicted scenes 
better suited outside of the classroom walls targeting the teen and adult population. Trends and themes which 
evolved were embedded into the game to reflect the repetitive needs expressed by children falling on deaf ears. For 
example, the name of the game changed to 4 Way Home as cases gathered continued to reflect the same underlying 
cry to be able to return to the home where they felt safe. 
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GPS: Navigate the Way! 


Chadren Around the Works Home Page: introduction 
ay riome Forth Teche 


About this Game 


Choose your adventure navigating through the following scenes to listen to the chitdren 


Choose Your eran rye 
Own Adventure Seonmnensentte suuti aioe 


See what lt would be like if you llved in Austrailia 
Take action! 


Alex is almost 18 months old now and can speak 


Begin here: Your Mission if you lived in Germany 


Visit the mom in Minnesota, USA 


If you get lost, use this roadmap to navigate the game: GPS 


Visit Jett in New York 
‘See how it is handled in some counties in California, USA 
Play to: Take me away to Costa Rica! 


-Identify specific human rights listed in the CRC treaty. 
-Articulate on Articles 12 and 13 of the treaty. oaphrae. betereras Ti 
-Apply those rights to real-world situations. 


-Identify which country has n ratified th reaty. x and Jim are 5 & 6 now 
fy untry has not yet ratified the CRC treaty. peoncraene yoo ae Pate Seana becrcene. 


Banglacesh THIS CRC 


Figure I. A Way Home: Introduction. Ensmann, 2017 


The game-for-social-change was designed to meet the gaps of the games upon which it was modeled. For 

example: 
iCivics teachers about the national rights of the US citizens per the national amendments. A Way 
Home teachers about the rights of children per the global CRC Treaty. 
Europa, uses exogenous games to teach children about the CRC in countries who have already adopted 
the Treaty. A Way Home was designed as an endogenous game, which means players would literally 
immerse themselves in scenes to experience the benefits of adopting the CRC in the US. 

Supportive tools are offered in every scene to provide guidance and feedback to players. For example, there 
is a CRC icon which lists the articles of the treaty, a GPS icon to help players navigate, and always the escape tunnel 
to offer players an outlet take control of the scenarios with access to real-world application inviting them to take 
action. Twinery, a web-based software was used, to design the storyline and create the intersections of the game. It 
started as a simple linear view and quickly blossomed to an impermeable web filled with numerous scenes from 
real-world cases. The prototype encompasses nine main characters, travelling and aging throughout the game, all 
eventually imploring the same need to be heard. 

The First level is a simple fictitious scene to introduce the player to the differences between rights and 
responsibilities, choices and consequences and tips to respectfully communicate with adults. The next level of the 
game, reflects real-world cases, such as the one which portrays the life of a Boy who hides his restraining order in 
his sock, or the story of the children who begged their mother for help. That family escaped and sought protection 
outside of the country. In the real-world case, they were awarded political asylum by another country (Collins, 
2012). 

It’s game over when players make decisions which end in incarceration or injury. One-hundred, forty-five 
children died last year just in Florida due to abuse or neglect (TCPalm, 2016). These are real endings in which real 
children lose. Players win the game when the children Win their Day to be heard. 
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Figure 2. A Way Home: You Win Scene. Ensmann, 2017. 


A way Home is a Game for Social Change. The purpose of this game is to 

e Introduces players to the CRC 

e Raise awareness of the need for adopting this treaty in the US and 

e Motivate players with actionable items 

Leading Learning for Change is not a solo affair, however. This game has been developed to the prototype 
stage. This researcher seeks out developers to take it to the next phase of development. Doctors Todres and. Diaz 
wrote in the Global Health journal just last month explaining why children's right to be heard matters: 

Participation can contribute to positive child and adolescent development. Studies show that allowing 


children to play a meaningful role in, and have some sense of control over, their lives have potentially significant 
intrinsic value (2017). 


Game-for-Social Change: “A Way Home” 


This game: 
. Introduces players to the CRC, specifically 
Article 12: The Right to be Heard. 


. Raise awareness of the need for adopting 
this treaty in the US. 

. Motivate players with actionable items, 
ultimately affecting change to improve 
equality for children everywhere. 


Figure 3. A Way Home: Purpose. Ensmann, 2017. 


Your contribution matters. Pass this information on or join the collaborative effort in pursuit of change to 
improve lives for children. 


165 


References 


Amar, A. R. (2012). America's unwritten constitution: The precedents and principles we live by. New York, NY: 
Basic Books. 

Assembly, U. G. (1989). Convention on the rights of the child. United Nations, Treaty Series, 1577(3). 

Australian Law Reform Commission. (1997). Seen and heard: Priority for children in the legal process. Retrieved 
March 14, 2017, from http://www.alrc.gov.au/sites/default/files/pdfs/publications/ALRC84.pdf 

Blow, J. (2004). Game development: Harder than you think. Queue, 1(10), 28. 

Breger, M. L. (2010). Against the Dilution of a Child's Voice in Court. Ind. Int'l & Comp. L. Rev., 20, 175. 

Brent, R., Felder, R. M. (2009). Active learning: An introduction. ASO Higher Education Brief, 2(4), 1-5. 

Bruch, C. S. (2001). Parental alienation syndrome and parental alienation: Getting it wrong in child custody 
cases. Family law quarterly, 35(3), 527-552. 

Centers for Disease Control and Prevention. (2014). FastStats - Births and natality. Retrieved March 23, 2016, from 
http://www.cdc.gov/nchs/fastats/births.htm 

Centers for Disease Control and Prevention. (2014). FastStats - Marriage and divorce. Retrieved March 23, 2016, 
from http://www.cdc.gov/nchs/fastats/marriage-divorce.htm 

Chen, S., Michael, D.R. (2006). Serious games: Games that educate, train and inform. Retrieved March 23, 2016, 
from http://books.google.nl/books?id=49kTAQAAIAAJ 

Cody, M., Ritterfeld, U., & Vorderer, P. (2009). Serious games: Mechanisms and effects. Routledge, Chicago. 

Collins, H. (2012). No way out but one. (DVD). Garland Waller Productions. 

Dormann, C., Whitson, J. R., & Neuvians, M. (2013). Once more with feeling: Game design patterns for learning in 
the affective domain. Games and Culture, 8(4), 215-237. 

Dormann, C., Whitson, J. R., & Neuvians, M. (2013). Once more with feeling: Game design patterns for learning in 
the affective domain. Games and Culture, 8(4), 215-237. 

Edele, A., Dziobek, I., & Keller, M. (2013). Explaining altruistic sharing in the dictator game: The role of affective 
empathy, cognitive empathy, and justice sensitivity. Learning and individual differences, 24, 96-102. 

Eichenbaum, A., Bavelier, D., & Green, C. S. (2014). Video games: Play that can do serious good. American 
Journal of Play, 7(1). 

EUROPA. (2011). EUROPA - Kids' corner. Retrieved December 13, 2015, from 
http://europa.eu/kids-corner/index_en.htm 

Gagne, R. M. (1981). Planning and authoring computer-assisted instruction lessons. Educational technology, 21(9), 
17-21. 

Gee, J. P. (2008). Video games and embodiment. Games and Culture, 3(3-4), 253-263. doi: 
10.1177/15554120083 17309 

Gordon, A. K. (1970). Games for growth; Educational games in the classroom. 

Gunter, G. A., Kenny, R. F. & Vick, E. H. (2008). Taking educational games seriously: using the RETAIN model to 
design endogenous fantasy into standalone educational games. Educational Technology Research and 
Development, 56(5-6), 511-537. 

Haddix, A. (1996). Unseen victims: Acknowledging the effects of domestic violence on children through statutory 
termination of parental rights. California Law Review, 757-815. 

Heeter, C., Lee, M. & Peng, W. (2010). The affects of a serious game on role-taking and willingness to help. 
Journal of Communication. 60(4), 723-742. 

Holt, S., Buckley, H., & Whelan, S. (2008). The impact of exposure to domestic violence on children and young 
people: A review of the literature. Child abuse & neglect, 32(8), 797-810. 

Huang, W. H. D. & Tettegah, S. (2010). Cognitive load and empathy in serious games: A conceptual 
framework. Gaming and cognition: Theories and practice from the learning sciences, 137-151. 

iCivics. (2011). Free lesson plans and games for learning civics. Retrieved November 22, 2015, from 
https://www.icivics.org/ 

iCivics. (2012). iCivics Partner Filament Games Wins Competitive Education Grants. Retrieved October 1, 2017, 
from https://www.icivics.org/ 

Jones, D. A. & Chang, M. (2014). Pecunia-A Life Simulation Game for Finance Education. Research & Practice in 
Technology Enhanced Learning, 9(1), 7-39. 

Jones, R. (2008). Saving worlds with videogame activism. Handbook of research on effective electronic gaming in 
education. London: Information Science Reference, 970-988. 

Justice, L. J. & Ritzhaupt, A. D. (2015). Identifying the barriers to games and simulations in education creating a 
valid and reliable survey. Journal of Educational Technology Systems, 44(1), 86-125. 


166 


Kapp, M.K. (2012). Can a video game make someone nice? eLearn Magazine. Retrieved February 26, 2015, from 
http://elearnmag.acm.org/featured.cfm?aid=2399242 

Keller, J. M. (2009). Motivational design for learning and performance: The ARCS model approach. Springer 
Science & Business Media. 

Lemon, N. K. (1999). The legal system's response to children exposed to domestic violence. The Future of Children, 

67-83. 

Merrill, M. D. (1997). Instructional strategies that teach. CBT Solutions (Nov/Dec), 1-11. 

Merrill, M. D. (2001). Components of instruction toward a theoretical tool for instructional design. /nstructional 

Science, 29(4-5), 291-310. 

Merrill, M. D. (2002a). First principles of instruction. Educational Technology Research and Development, 50(3), 


43-59, 

Merrill, M. D. (2002b). A pebble-in-the-pond model for instructional design. Performance Improvement, 41(7), 39- 
44. 

Merrill, M. D. (2007). A task-centered instructional strategy. Journal of Research on Technology in Education, 
40(1), 33-50. 


Merrill, M. D. (2007). First principles of instruction: a synthesis. In R. A. Reiser & J. V. Dempsey (Eds.), Trends 
and Issues in Instructional Design and Technology, 2nd Edition (Vol. 2, pp. 62-71). Upper Saddle River, 
NJ: Merrill/Prentice Hall. 

Merrill, M. D. (2009). First principles of instruction. In C. M. Reigeluth & A. Carr (Eds.), Instructional Design 

Theories and Models: Building a Common Knowledge Base (Vol. 3). New York: Routledge Publishers. 

Merrill, M. D. (2009). Finding e3 (effective, efficient and engaging) instruction. Educational Technology, 49(3), 15- 

26. 

Moriarty, J. E. (1974). Simulation and gaming: Proceedings of the 12th annual symposium, National Gaming 
Council, and the 4th annual conference, international simulation and gaming association, held at the 
national bureau of standards, Gaithersburg, Md., September 17-19, 1973 (Vol. 395). National Bureau of 
Standards. 

Packard, E. (1979). The cave of time. Choose your own adventure. 

Pellegrino, J. & Scott, A. (2004). The transition from simulation to game-based learning. In The 
Interservice/Industry Training, Simulation & Education Conference (I/ITSEC), NTSA. 

Prensky, M. (2001). Digital natives, digital immigrants part 1. On the horizon, 9(5), 1-6. 

Prensky, M. (2006). Don't bother me, Mom, I'm learning!: How computer and video games are preparing your kids 
for 21st century success and how you can help!. St. Paul, MN: Paragon house. 

Ratan, R., Ritterfeld, U. (2009). Classifying serious games. Serious Games: Mechanisms and Effects, 10-24. 

Reigeluth, C. M., Merrill, M. D., Wilson, B. G., & Spiller, R. T. (1980). The elaboration theory of instruction: A 
model for sequencing and synthesizing instruction. Instructional Science, 9(3), 195-219. 

Roberts, G. G. (2002). SET for success: the supply of people with science, technology, engineering and mathematics 
skills: The report of Sir Gareth Roberts' review. Retrieved April 6, 2017, from 
http://webarchive.nationalarchives.gov.uk/+/http:/www.hm-treasury. gov.uk/d/robertsreview_introch1.pdf 

Rodham, H. (1973). Children under the law. Harvard Educational Review, 43(4), 487-514. 

Rogers, E. M. (2010). Diffusion of innovations. Hohenheim, Germany. Simon and Schuster. 

Taylor, L. (2009). A lawyer for every child: Client-directed representation in dependency cases. Family Court 
Review, 47(4), 605-633. 

TCPalm. (2016). CASTLE’s Memory Field Ceremony in Stuart. TCPalm News: Part of the USA Today Network. 
July 29, 2017, from http://www.tcpalm.com/picture-gallery/news/2016/07/29/CASTLEs-Memory-Field- 
Ceremony-in-Stuart/87835976/. 

TechKnowledge. (2009). Gaming for change - G4C fest puts serious games to the test. School Library Journal, 8. 

Todres, J., & Diaz, A. (2017). Adolescents' Right to Participate: Opportunities and Challenges for Health Care 
Professionals. Annals of Global Health. 

Toppo, G. (2015). Sandra Day O'Connor's post-court legacy. Civics games. USA Today. Retrieved April 28, 2015, 
from http://www.usatoday.com/story/news/2015/04/14/oconnor-civics-games-legacy/25505871/ 

Treaties. (2014). Convention on the rights of the child - United Nations. Retrieved February 6, 2016, from 
https://treaties.un.org/Pages/ViewDetails.aspx?src=IND&mtdsg_no=IV-11 &chapter=4&lang=en 

UNICEF. (2009). State of the world's children: Celebrating 20 years of the convention on the rights of the child. 

UNICEF. 


167 


STEAM Powered Tools For Art Education 


David Gardner 


Colby Parsons 


Abstract 


This paper seeks to highlight an effort by Computer Science and Visual Arts faculty to create opportunities 
for students from both departments to work with interactive technology in cross-disciplinary creative teams. The 
focus of the paper is a group project assigned within a STEAM course co-taught by faculty from both disciplines. 
The project, which challenges students in the course to build interactive educational tools for use in K-12 art 
education, will be presented in terms of its pedagogical role as a “bridging” assignment unifying the disciplines, and 
also tying together the skill acquisition and the skill application portion of the course. 


Introduction 


Course creation in the field of Interactive Digital Art can present challenges for both designers and 
instructors, since it draws skills from the very different disciplines of Visual Art and Computer Science. It is 
particularly challenging to bring together students with experience limited to one discipline or the other and 
effectively engage and teach both groups simultaneously; particularly if the goal is to leave them empowered to 
continue exploring the subject beyond the end of the semester. To address the issue, the course creators developed a 
project that tasks the students with working collaboratively to create educational tools for K-12 art classes. This 
project was created so that students would learn elements from both disciplines while creating an attainable result 
that is also useful and fun. 


Foundations 


The highlighted project is situated within a course designed and co-taught as a collaborative effort between 
faculty in computer science and visual arts. The intention of the course is to have students explore topics in 
Emerging Media and Interactive and Generative Art, topics that require a range of skills not commonly found in 
students from either of the academic disciplines. An additional goal of the course is for students to gain insight and 
experience regarding how to work collaboratively with individuals from outside their own discipline in creative 
teams. The fact that the course is co-taught by both computer science and visual arts faculty means that variations in 
perspective, logic, and creative approach can be continually addressed and discussed. As such, the teaching process 
also models the cross-disciplinary collaborative process the faculty found necessary in order to communicate ideas, 
provide instruction, and create sample projects. 

Throughout the course, students work individually and in small groups to explore programming, animation, 
electrical circuits, input/output devices, and interface construction. Processing, a Java based framework designed to 
make it easy to programmatically create animation and many other results, is the primary visual creation tool used in 
the course. That programming language was chosen because it provides an easy entry to programming for artists and 
a means for the computer scientists to learn more about artistic concepts. The Arduino microcontroller board, and 
devices such as buttons, motors, lights, and dials, are used to teach students the basics of electrical circuits for the 
purpose of building unique interfaces to control visuals, animation, audio, and other results. 


Problem 


Part of the struggle for the instructors in the previous iterations of the course has been balancing the 
approaches and content being presented. The course brings together students from different disciplines which poses 
interesting obstacles for course creators. These obstacles go beyond a difference in skill sets and include issues 
related to general temperament, expectations of course assignment, timing, and grading, as well as the context 
within which learning should occur. In general, the art students enrolling in the class tended to adopt a more flexible 
and open approach to tasks and tended to have some misgivings about technology. These students brought a 
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willingness to see activities as continual processes and to be a bit wary of structure. They were used to three hour 
long classes twice a week, more subjective evaluation, and varied workspaces. The computer science students on 
the other hand favored structure and clearly defined processes and goals. They tended to struggle with tasks that 
lacked a defined functional end and were used to a traditional computer lab as the space in which learning occurred. 
Furthermore, they were used to classes that were half as long as those experienced by art students and expected a 
more objective approach to evaluation. They had the expectation that If they adhered to the constraints of the 
assignment, then that should result in a perfect score on any assignment. 

Course creators had intended for their to be a balance between these different outlooks, but in early 
iterations of the course instructors struggled to move beyond the need to teach technical abilities in a way that would 
help students develop projects grounded in artistic concepts. This technical focus, especially in the early weeks of 
the class, left the art students feeling uncomfortable with the programming heavy content and functional focus of 
tasks. Other assignments where students were required to apply the technical skills to communicate an artistic 
concept left computer science students struggling to find a functional purpose behind the project. Group projects in 
the early iterations of the course were purposefully designed to be open-ended in order to allow the students to apply 
the wide variety of hardware, software, and technical processes in any way they wished to achieve an articulated 
artistic concept. Unfortunately, this broad amount of opportunity left both art and computer science students 
overwhelmed and struggling to define a path forward in the projects. 

To an extent this disconnect in student perspectives is to be expected based on the nature of the course, but 
the instructors sought a way to create a more integrated approach that would more effectively unify these divergent 
outlooks. Toward this end, a project was designed to simultaneously provide a functional focus for the computer 
science students to learn the artistic concepts and an artistic grounding that allowed the art students to incorporate 
technical elements. It was decided that this project would act as abridge” to address the issues in discipline 
expectations. Furthermore, the project would tie the early weeks of the course which were focused on small skill 
acquisition assignments to the later weeks of the course which were focused on larger concept focused interactive art 
installations. The solution they landed upon was the “Color Theory Project” described below. 


Solution 


The Color Theory Project was created to address the problems of contextualizing knowledge for both 
groups of students. The intention was to do so in a way that allowed them to teach each other by building tools for a 
specific need, but in a context that seems achievable despite limited experience. Students in the course were 
grouped into teams of three, consisting of one artist, one junior computer scientist, and one senior computer 
scientist. Each group was tasked with creating an interactive visual meant to help K-12 students learn a specific 
aspect of color theory. 

Each group was given a different color theory concept as the focus of their project and then needed to 
design both a screen based interactive visual and a physical interface for controlling the visuals. Students were 
asked to include multimedia elements and to explore ways to teach the concepts to children in a manner that was fun 
and interesting. The expertise of Art Education specialists was sought to determine what artistic elements K-12 
students struggle with at various grade levels and what content goals an instructor would want to achieve through 
the use of such a tool. 


Four groups were created within the class and were randomly assigned the following color concepts to 
address in their creations: 
- Color Harmony 
- Color & Contrast 
- Additive & Subtractive Color 
- Color Context 


Core requirements for the project included: 

- An initial segment or screen that presents this color theory concept, and indicates a way to move on to a 
second screen. 

- A second segment that challenges the student to apply the color theory concept (and gives feedback 
regarding whether they have applied it correctly). 

- A clearly defined end that resets to the first screen. 

- A physical interface constructed of cardboard and incorporating arcade buttons or other input devices. 
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Students were given time in class and were also required to work outside of class to create the project. The 
projects were to be completed in a two-phase process wherein the first phase focused on the creation of the visuals, 
content, and gameplay using the Processing language. This phase of the project covered roughly two and a half 
weeks and students were required to provide weekly updates both to the instructors and the rest of class on the 
current progress of the project including the successes and issues they had experienced during the creation process. 
This was done in order to allow the entire class to benefit from any solutions to specific problems other groups had 
come across and to allow all students to suggest solutions to existing problems in the development of the projects. 
This activity also allowed the instructors to offer suggestions and address any issues within the creative teams or re- 
focus the students on their assigned color-theory concept as needed. The first phase culminated in a class-based 
demonstration of the visual portion of the project and an opportunity for instructors and students to try out the 
current state of the project and offer suggestions on revisions. Art Education majors also attended this 
demonstration to give feedback on appropriateness and the overall feel of the visuals. 

A gap between the first and second phase of the project occurred as students worked on small assignments 
related to soldering, wiring, and use of electronics as controlled by the Arduino microcontroller. During the gap 
between the two phases, students began the ideation process for their final group projects and began acquiring 
resources to complete this semester project. Before the gap in work on the Color Theory Project, students were told 
that they would need to be working on a physical interface for the visual element of the Color Theory Project 
consisting of a painted/decorated cardboard-based prototype enclosure integrating functioning arcade buttons wired 
to an Arduino. They also had to set up the Arduino to accept those signals and pass them along to the Processing 
code running on the computer. Students were informed that they would be presenting this final phase of the project 
to the public at a Creative Arts & Research Symposium held at the university where the course is taught. All groups 
would have the week before the symposium to finalize their physical interfaces but they would need to be working 
on this second phase of the project in the background while the class focused on other related activities concerning 
the Arduino. 

The second phase of the project concluded with the various creative teams presenting their projects for use 
by the public at the symposium. A large room that normally held tables and chairs for approximately 40 individuals 
was the context for this public showing and tables and chairs were rearranged in the room to allow the public to 
move around the room where the various projects were set up. Student projects were presented in various states of 
functionality. The majority of projects’ physical interfaces consisted of enclosures that masked the laptop 
computers (driving the visuals) as well as the wiring and Arduino (controlling physical input). The only parts of the 
project that were visible to the public In these creations were a screen, the tops of arcade buttons, and the enclosure 
covering. Some projects relied on constructed coverings for a traditional mouse as well as button based controls to 
interact with the visual elements of the project. 


Results 


Overall the majority of the groups were able to complete a functional version of the Color Theory Project 
that included a game based visual tutorial for the assigned color concept and a prototype physical interface. One 
group was not able to get a functioning physical interface ready for the symposium and resorted to the standard 
keyboard and touchpad of a laptop to control the visuals. At the symposium the projects were well received and 
several attendees encouraged others in different venues of the symposium to come and interact with the projects. 
Some attendees mentioned that they enjoyed trying out the creations and felt that the projects might work well in a 
classroom when advanced to a more solid physical interface of plastic or other material. Students reported feeling 
satisfied with the knowledge gained about the various technologies and the experience of developing prototypes. 
The instructors felt that the experience did advance the students’ abilities to work with Processing and Arduino 
microcontrollers. However, the main benefit of the project within the course was to simply allow students to 
practice and apply skills related to the use of the technology and it did not fully meet the goals of the creators. 

While some of the physical interfaces reached a generally complete state, the groups struggled to create 
physical interfaces beyond those which masked and mimicked the normal functions provided by a computer. The 
creators had hoped that the students would exercise some creativity in exploring different ways of controlling the 
visuals, but the interfaces generated by the students tended toward the purely functional. Many students 
underestimated the time necessary to create and troubleshoot the physical interfaces, and thus resorted to the easiest 
path to completion. This in part may be due to the functional focus of many of the computer scientists who largely 
maintained directive control over the projects and adopted a “meet the base requirements” approach to the project. 

It was hoped that the content of the project being focused on a formal element of art would allow art 
students to see an avenue into the project that would allow them to direct the group creatively and generate 
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interesting and aesthetically appropriate projects. However, this did not happen. The art students tended to struggle 
to find a role for themselves in the project and some fixated on a specific artistic element regardless of its relevance 
to the project or ability to easily integrate into the concept or gameplay. Others simply separated themselves from 
the process and focused on other assignments. This disconnect may in part be due to an unconscious assumption by 
the creators that the art students would naturally fill the role of creative director. In hindsight, that supposition on 
the part of the creators failed to take into account the tendency among art students to work on their own. 
Furthermore, this disconnect may be the result of the project being too focused on a specific task which might be 
considered rudimentary from an artistic perspective. The art students may have felt that they did not see a way that 
they could express their own creative intentions within the context given. 

Additionally, many of the created projects were vague in how they tested the color concepts they were 
assigned to address. Often there were contextual choices that confused or distracted from the main concept. The 
computer scientists often became overwhelmed with the possibilities for gameplay and struggled to achieve some of 
the technical requirements programmatically. They seemed to adopt a “good enough” attitude and took visual 
shortcuts because it was more expedient. Image choice and programmatic structure were often chosen because of 
easy availability rather than because of conceptual appropriateness. While the project provided a clear functional 
goal for the computer science students and an artistic component for the art students, the project as a whole was 
more utilitarian than conceptual. The technical difficulties prevented the groups from reaching a point where they 
could explore alternative iterations, and a lack of opportunity for creative expression seems to have resulted in 
diminished morale. 


Discussion 


Ultimately, the course creators felt that the Color Theory Project did not meet the needs it was designed to 
address. This is not to say that the project did not have value, but the value of the project in this course was more as 
an opportunity for students to learn the value of iterative development. The course creators feel that the project was 
ineffective in supporting the development of artistic concepts. The project offered insight into the assumed abilities 
of students and the need for projects based on prepared technical solutions that allow students to quickly reach a 
level where they can explore artistic concepts. This is not to say that the course creators seek to remove all technical 
difficulty from future projects, but rather they feel that the are enough inherent difficulties in implementing and 
adapting prepared solutions to unique situations to be sufficient for learning. In essence, the goal will be to 
bootstrap the students up to a point where can achieve initial success in specific applications before branching out to 
more self-directed interactive digital art applications for the final project and beyond the course. 

Furthermore, while the Color Theory Project will probably not be included in future iterations of the 
Interactive Digital Art course, it has been the potential to be the basis for development of other inter-disciplinary 
efforts. Specifically discussions are underway to form a partnership between computer science and education that 
stems from the example set by the course for which the project was originally developed. This sort of partnership 
provides a better context for the Color Theory Project and the opportunity for expansion and could potentially 
benefit from more motivation based on authentic interest from students. 
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Abstract 


Wearable computers include a variety of body-borne sensory, communication, and computational 
components that may be worn on the body, under, over, or within clothing. These mechanisms have potential 
benefits for (a) human performance support, (b) cognitive and psychomotor learning, and (c) K-12 educational 
environments. This manuscript begins with a historical overview of wearable computers and then provides the 
readers with a current and future perspective of wearable use across a variety of educational environments with 
specific attention focused on K-12 environments. Examples of current research in wearables is highlighted, followed 


by limitations and future directions for research. 


Introduction and Background 


Wearable technologies have progressed over the past decade and have the potential to be used effectively in 
K-12 classrooms (Lee, Drake, & Williamson, 2015). Wearable computers have been around since the early 1960s 
(McCann & Bryson, 2009). Thorp and Shannon created a roulette wheel predictor, a wearable computer that would 
predict where the ball would land when playing roulette (McCann & Bryson, 2009). The device did not earn the title 
of the first wearable computer until 1966 when Thorp published the work (McCann & Bryson, 2009). Another 
contributor to wearable computers was Steve Mann. Mann developed his first wearable computer system in the early 
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1980s, and it was composed of a head-mounted camera and a backpack (Mann, 1997; McCann & Bryson, 2009). 
Over the next 20 years, Mann’s (1997) wearable computer continued to evolve into a less cumbersome device. 

In the late 1980s and early 1990s, further progress in the area of wearable technology made smart glasses 
commercially available (Havard & Podsiad, 2017). After the introduction of the World Wide Web, researchers 
began sharing their wearable computing studies internationally (McCann & Bryson, 2009). The sharing of 
information enabled technology developers to combine the ideas of multiple wearable computers to create new 
technologies. In 1993, Platt and Starner combined smart glasses called the Private Eye and a one-handed keyboard 
to develop the first context-aware system (Havard & Podsiad, 2017). Throughout the 1990s, researchers developed 
additional wearable computers such as the Pathfinder system, which was the first wearable GPS, as well as 
prototypes for augmented reality systems (Havard & Podsiad, 2017). 

In the early 2000s, the Lilypad Arduino was introduced and began as an academic research project 
(Buechley & Hill, 2010). “The LilyPad Arduino is a system for experimenting with embedded computation that 
allows users to build their own soft wearables by sewing fabric-mounted microcontroller, sensor and actuator 
modules together with conductive thread” (Buechley, Eisenberg, Catchen, & Crockett, 2008, p. 424). The Lilypad 
project was commercialized in collaboration with Sparkfun Electronics and sold as an e-textile construction kit 
(Buechley & Hill, 2010). Students in K-12 have used the Lilypad Arduino to make a “touch-sensitive shirt; makes 
silly sounds when touched in certain places and a police hat that makes siren noises when a switch is pressed” 
(Buechley & Eisenberg, 2008, p. 14). The Lilypad Arduino serves as the electronic component of many wearable 
devices. 

Although advances in wearable technology have progressed significantly over the past few decades, 
researchers and educators are continually developing wearable devices and finding new ways to incorporate them 
into the academic curriculum. Currently, wearable devices are being used in a variety of ways in K-12 education. 
Educators and researchers across the globe have infused fitness activity trackers in schools to help students achieve 
instructional goals. Additionally, wearable technologies have been paired with STEM instruction and educational 
computing, as well as a tool to engage students in collaborative learning experiences. Wearable devices are also 
being used with students to encourage educational gaming and free play. Though there are limitations to the use of 
wearable devices, these technologies have positive implications for teachers and students alike. 


Wearable Fitness Activity Tracking Devices 


The introduction of wearables for health-related purposes was not until the 1980s (Price & Rasmussen, 
1980). Price and Rasmussen (1980) patented the invention of a wearable heart rate monitor for the wrist that 
detected and displayed one’s pulse rate. The technology has evolved to fitness and activity trackers, smartwatches, 
and heart-rate monitor chest straps. More recently activity tracker wearables have been used in K-12 classrooms for 
educational purposes besides teaching students about health or fitness. 


Statistics Instruction 


High school students have used Garmin Forerunner heart rate monitors to practice interpreting visual 
displays of data, and fifth-grade students have used Fitbit Ultra device data to identify measures of central tendency 
(Lee et al., 2015). Lee, Drake, and Thayne (2016) chose to utilize Fitbit Ultra and Fitbit One devices during 
physical education and teach students grades 3-8 elementary statistics. Steps gathered from the students’ activity 
trackers were used as data to create histograms and provide students with an understanding of variability (Lee et al., 
2016). Fitbit is a pedometer, or step counter, that uses a “three-axis accelerometer to detect movement” (Lee et al., 
2016, p. 357). The wearables were not used to teach students specifically about physical education but were used to 
teach students about data accuracy and statistics (Lee et al., 2015). 


Project GETUP 


Project GETUP (Gaming to Educate Teens to Understand Personal Health) was a study conducted to 
determine student engagement with tracking his/her health using Fitbit One devices (Schaefer, Ching, Breen, & 
German, 2016). Schaefer et al. (2016) discovered that student engagement declined over time, tried to cheat to log 
more steps, and there were several constraints that could have possibly affected the outcome of the data. The 
potential constraints included limited technology accessibility, design flaws, difficulty using the device, and device 
loss (Schaefer et al., 2016). 
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Motivational Tool 


At the A. Harry Moore School at New Jersey City University, Fitbits are used to motivate students to move 
around during the school day (Pepe & Talalai, 2016). A. Harry Moore School is a laboratory school for “students 
with multiple physical, medical, and cognitive disabilities” (New Jersey City University, n.d., para. 1). The Fitbit 
serves as an activity tracker and steps counter display that aids in encouraging students to meet daily goals (Pepe & 
Talalai, 2016). 


Wearables for Instructional Needs 


The use of wearable technology to meet instructional goals in the K-12 environment is occurring across the 
globe. Instructors are incorporating e-textiles and other wearable technology into instruction to improve attitudes 
and interest in STEM and engineering, as well as educational computing. Building and programming wearables 
encourage creativity and facilitate cooperative learning among K-12 students (Ngai, Chan, Cheung, & Lau, 2009). 


Science, Technology, Mathematics, Engineering (STEM) Instruction 


Solving real-world problems with the use of the engineering design process is the cornerstone of STEM 
instruction. Learning through engineering design lessons allow students to connect content knowledge to real-world 
applications (Riskowski, Todd, Wee, Dark, & Harbor, 2009). Therefore, researchers have attempted to discover the 
impact of wearable technology and engineering design. WearTec researchers at the University of Nebraska have 
found evidence that intermediate students who participate in wearable technology programs have increased attitudes 
towards STEM, including motivation to learn, self-efficacy and learning as a whole (Barker et al., 2015). WearTec 
researchers also found that the use of e-textiles in instruction has been shown to increase interest and participation in 
female students because it makes engineering and computing personally relevant to them (Barker et al., 2015). In 
addition, Barker et al. (2015) stated that the instructional goals of wearable technology are closely related to the 
goals of the engineering design process; therefore, making the use of these technologies a natural addition to STEM 
education. 


Collaborative Learning 


Also in line with the engineering design process, the use of wearable technologies promotes collaborative 
learning with K-12 students (Ngai et al., 2009). Researchers using e-textiles for computing circuit design observed 
several students working on different aspects of one garment at the same time, recognizing the need to work 
simultaneously with other students to accomplish their goal (Ngai et al., 2009). Even students attending different 
schools have the ability to collaborate with to create and produce wearable technologies. Middle and high school 
students working with the Engineering Brightness project used 3D printers to develop wearable wrist watches with 
solar-powered lights so children in underdeveloped countries can read at night without electricity (Fogarty, Winey, 
Howe, Hancox, & Whyley, 2016). Using online conferencing applications, such as Skype, students worked 
collaboratively to learn necessary information on circuits, plan and design the wearable wrist watches, deepening 
their understanding of the technology, as well as its philanthropic impacts (Fogarty et al., 2016). 


Educational Computing 


Wearable technology gives K-12 students an opportunity to master instructional goals, as well as designing 
and creating projects using educational computing (Barker et al., 2015). Although wearable technology shows 
promise in the area of educational computing, it comes with some significant challenges that researchers are aiming 
to improve. Researchers created the TeeBoard in an attempt to utilize e-textiles in a way that allows students to 
make mistakes that are easily correctable and does not require extensive training or expensive tools (Ngai et al., 
2009). The TeeBoard project is just one example of how e-textiles and other wearable technology permeate the K-12 
curriculum. 


Wearables for Games and Other Uses 


Researchers have found that incorporating wearable technology into play time fosters creativity in young 
children, encourages physical activity and allows students to play independently (Rosales, Sayago, & Blat, 2015). 
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Wearable technology has also been piloted in classrooms to assist students with hearing impairments with the use of 
Google Glass and Quick Response (QR) codes. 


Gaming and Play 


Not only does wearable technology spark interest for young children, it may also serve a valuable purpose 
when playing games. The creators of the BeeSim game used e-textile puppets with students ages 7-8 to illustrate 
how complex systems operate by using honeybees as a participatory simulation (Peppler et al., 2010). Students used 
their computational puppets to collect honey and communicate with other students acting as bees, the beehive, and 
flowers. Although students attempt to win the game by bringing the most honey to the beehive, they also realize the 
importance of working quickly and communicating with the other bees (Peppler et al., 2010). 

Researchers facilitated a workshop with middle school students that encouraged them to creatively modify 
existing games to include wearable controllers (Vasudevan, Kafai, & Yang, 2015). Students used computational 
construction kids to design and create wearable controllers that coincide with the Flappy Birds computer game. The 
children that participated in the workshop were encouraged to be creative; therefore, each computational glove 
looked slightly different and codes varied from student to student (Vasudevan et al., 2015). This workshop provided 
students with an opportunity to participate in the creation of the controller, as well as the activity of testing and 
playing with their final product. 

Other technologies such as Wearable Sounds, Statue and FeetUp are wearable accessories that were created 
for use during free play for young children (Rosales et al., 2015). The FeetUp accessories encourage movement, as 
they only chirp when both feet are off the ground (Rosales et al., 2015). In a similar fashion as those listed above, 
these wearable accessories allow students to express themselves creatively through play, sometimes creating new 
and alternative games to those suggested by the researchers (Rosales et al., 2015). 


Assistive Technology 


Students with significant hearing impairment benefit from the development of Glass Vision 3D which uses 
a Google Glass application for assistance in the classroom. For this project, Google Glass was used in conjunction 
with QR codes, allowing students to scan the codes with their glasses which prompts an American Sign Language 
video to appear via augmented reality (Parton, 2017). Students are also able to gesture to access videos on the 
glasses, rather than use their voice because many students with hearing impairments are not comfortable with verbal 
language (Parton, 2017). 


Limitations 


Although the use of wearable technology in the K-12 environment yields positive outcomes in many 
occasions, researchers and educators have also determined that there are significant concerns to address in the 
future. Researchers noted that using Google Glass for over one hour caused the device to overheat. Therefore, the 
students had to wait until the glasses were cool before further use (Parton, 2017). Classroom teachers stated that 
although wearable technologies sparked interest for their students, instruction took significantly longer than other 
technologies, such as iPads, even though the outcome was similar (Parton, 2017). 

Researchers have also acknowledged that not all schools can afford Fitbits or wearable technologies for 
students (Lee et al., 2016). Schaefer et al. (2016) noted that it was “difficult to obtain all of the necessary 
technological resources” (p. 13) to syne wearable fitness devices in their urban afterschool program. Researchers 
discovered the limitations with school firewalls when uploading Fitbit data to the online website (Lee et al., 2015). 
Additionally, students may have limited access to the internet at home to sync wearable device data to online 
locations (Schaefer et al., 2016). It is possible for school technology specialists to set up temporary accessibility to 
provide students with online access to their Fitbit data in an effort to combat this challenge. Other limitations 
included privacy and ethical concerns, data storage, and data displays (Lee et al., 2015). 

As researchers conduct studies in K-12 settings, they noted small sample size as a significant barrier in 
generalizing their findings to a larger population. Barker et al. (2015) stated that because their sample size consisted 
of only 21 participants in the WearTec study, “the results cannot be generalized to the target population as a whole” 
(p. 74). These concerns were also echoed by Ngai et al. (2009) who indicated that the sample size of 25 was a 
limitation during their project. Ngai et al. (2009) stated it is “crucial that it be feasible to run larger-sized 
workshops” with qualified instructors (p. 56). Additionally, researchers noted that they wished to not only expand 
their sample size but also with participants of varying age groups. In Rosales et al. (2015) study, the authors 
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described their desire to test wearable devices with teenagers and adults who they thought “could also benefit from 
wearables that support their interest in play and social interaction through technology” (p.47). Regardless of 
barriers, it is evident that researchers have an overwhelming desire to continue and expand studies of wearable 
devices for educational purposes. 


Conclusion and Future Directions 


Over time, wearable technologies have continued to advance in both quality and quantity of features 
offered for consumers, as well as students in K-12 environments (Lindberg, Seo, & Laine, 2016). Researchers and 
educators alike are creating and presenting wearable computers that are user-friendly and guide students through 
basic computing functions (Ngai, Chan, Cheung, & Lau, 2010). With future research, wearables may be easier to 
incorporate into instruction, allowing both students and teachers the ability to reuse and reprogram interfaces so that 
they can be modified for other instruction and scaffolded for various concepts in the curriculum (Peppler et al., 
2010). Creating wearables such as e-textiles enable students to participate in tactile learning that supports child 
development and free play (Rosales et al., 2015). Additionally, wearable technologies allow students to learn 
creatively through the use of STEM disciplines and the engineering design process, which encourages success in 
higher education (Riskowski et al., 2009). 

Future studies may consider training educators in wearable technologies and their implementation to 
discover how wearables can be incorporated into science, technology, engineering and mathematics curriculum and 
instruction (Barker et al., 2015). Limited technology access in some communities may present a challenge when 
integrating wearables into the K-12 environment (Schaefer et al., 2016). Adapting wearables into education would 
require a smooth transition by incorporating these technologies a little at a time to avoid a backlash if some 
technologies do not work as effectively as initially anticipated (Borthwick, Anderson, Finsness, & Foulger, 2015). 

It is clear through the various studies discussed above that the analysis of data “captured through wearable 
technologies and the Internet of Things represents an invaluable source of information” for students and instructors 
alike (de la Guia Cantero et al., 2016, p. 377). Since wearable technologies allow students and teachers to monitor 
their actions while collecting data, they have the ability to look at data from a new, more personal perspective (Lee 
et al., 2015). Instructional time will essentially be saved in the long run due to quick and efficient data collection 
using wearables, although additional time may be spent initially learning how to operate these devices (Lee et al., 
2015). Based on the current research, students in the K-12 environment are not only benefiting from the use of 
wearable technologies in the classroom, but they are also open to their use in coordination with other learning tools 
and strategies (ul Amin, Inayat, & Shazad, 2015). 
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Design and Development of a Tool to determine E-learning Readiness 


Dr. Cathy James-Springer 


Dr. Katherine Cennamo 


Introduction 


Why does an organization decide to use e-learning in their workforce training? What are the potential 
challenges ahead? As e-learning becomes more commonplace as a tool for workforce training, it is recognized there 
are varying reasons for using e-learning, chief among them being the fear of losing competitiveness (Comacchio & 
Scapolan, 2004). Adoption for the sole purpose of remaining competitive does not necessarily result in the most 
appropriate and timely interventions. As training expenditures on e-learning continue to grow, there are increased 
advantages for adoption among which are the ability to reach geographically dispersed employees; deliver just in 
time, standardized training; and reduce costs associated with training. Since the use of e-learning is growing it 
becomes increasingly important to determine the readiness of organizations for adoption before and even after using 
e-learning. 

In this paper we will focus on the design and development process used in the creation of a tool to 
determine e-learning readiness. It begins with the initial considerations for tool development to the factors that 
influence the decisions made in the design and development process. The paper also outlines some of the limitations 
encountered in the research journey that determined the methodology, design and final development of the tool. 

Initial consideration in designing this tool was to determine from the onset what type of analysis would be 
necessary to determine e-learning readiness. Tessmer (1990) recommends two areas for consideration when 
designing instructional systems; the instructional environment and the support factors influencing the instructional 
environment. In undertaking an e-learning project some sort of needs analysis should be undertaken. Tessmer (1990) 
emphasizes the need for conducting an environmental analysis early in the project design since an understanding of 
the environment could guide later decisions. The environment refers not only to the physical environment but also 
related culture and climate. From an instructional design perspective, a learner and context needs analysis is usually 
included in course design, but e-learning represents an institutional-wide learning initiative and therefore a more 
encompassing analysis is needed. 

This research is based on the premise that a type of needs analysis is necessary before establishing e- 
learning within an organization. Having established the need for an upfront analysis, four (4) existing models for e- 
learning readiness were explored (Aydin & Tasci, 2005; Borotis & Poulymenakou, 2004; Chapnick, 2000; 
Psycharis, 2005) as examples of types of analysis tools. The four models included several similarities and variations 
in the aspects of e-learning that were important for consideration before establishing e-learning. A breakdown of 
each tool is seen in Table 1. 


178 


Table 1. Summary of E-learning readiness tools 


Aydin & Tasci (2005) Chapnick (2000) Borotis and Psycharis (2005) 
Poulymenakou 
(2004) 
1. Technology 1. Psychological 1. Business 1. Resources 
- Access 2. Technical 2. Technology - Technology 
- Learner skills and Readiness 3. Content - Financial 
attitudes 3. Environmental 4. Training process - Human 
2. Innovation 4. Human resources 5. Culture Resources 
- Barriers 5. Sociological 6. Hunan Resources | 2. Education 
- Adoption 6. Economic 7. Financial - Content 
- Openness 7. Financial - Educational 
3. People 8. Equipment 3. Environment 
- Employees 9. Content -  Entrepreneuria 
- HR Specialists 1 
- E-learning champion - Culture 
- Vendors and external - Leadership 
parties 
- Skills with respect to 
learning with 
technology 
4. Self-Development 
- Budget 
- Time management 
- Belief in self- 
development 


Source: p. 16, James-Springer (2016). 


Every tool referred to the need to consider: organizational readiness in the categories of organizational 
culture, human resources, and financial resources; learner readiness in areas such as learner characteristicsequipment 
readiness that covers infrastructure as well as readiness with reference to the operation of the equipment. For these 
tools, similarities were identified in broad headings, but there were variations in how the authors interpreted the 
categories within these headings; for example what Aydin and Tasci (2005) classify as self-development what can 
be easily categorized as learner self- efficacy and motivation, this same concept would fall under human resources in 
Chapnick’s (2000) and Borotis’s and Poulymenakou’s (2004) models. These variances provided a basis for the 
designing and developing this new tool for e-learning readiness. 


Methodology 
The study uses Design and Development (D&D) research methods that cross both model development and 


tool development, as defined by Richey and Klien (2007). A summary of the process and their resulting outputs are 
shown in figure 2. 
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Resulting Output at each stage 


process 


*Establishing e-learning readiness factors 


*Establishment of themes to be developed 


*E-learning readiness tool which is 
eSystematic in data collection 
Design and | eable to capture information on the environmental context 
Development! sFasy to use and report data 


Formative Evaluation of the content and construct of the tool 


Figure 2. Stages in the design and development process and resulting outputs 


In this research, two phases of analysis occurred, and the results of these analyses informed the design and 
development decisions. 

Analysis Phase | involved a review of four existing e-learning readiness models, chosen on the basis of 
their frequency of use in e-learning literature, which resulted in a preliminary list of factors required for e-learning 
readiness. 

Analysis Phase 2 involved a review of multidisciplinary literature to create themes. The literature used in 
this phase evolved as a result of questions raised during the preliminary review of the e-learning models. 
Nunamaker, Chen, and Purdin (1990) recommend that in information systems development, it is necessary to go 
beyond the discipline to obtain ideas for the design. Exploring other literature provided insight into ways of varying 
the tool, and facilitated application of content from fields such as learning and innovation principles within 
organizations, knowledge management, adult learning theory and technological support systems. These areas viewed 
in isolation have little in common with e-learning but the concepts proved useful in bringing the various aspects 
required for determining e-learning readiness. 


Design and Development 


The second phase involved the application and embedding of theoretical knowledge (Van den Akker et al. 
,2012) in the design and development of a tool that was practical and usable in real world settings. The tool was 
designed and developed to capture information on: 

Specific parameters for determining e-learning readiness 

Systematic processes for coming to conclusions on readiness 

Capturing concerns about the environmental context 

Interrelatedness of the overall implementation process 
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Development 


The tool was developed to incorporate factors identified as important in existing e-learning literature. 
Seventy six (76) sources of literature were examined and coded based on the themes identified in the second 
analysis phase. These comprised empirical qualitative and quantitative studies, business publications, and books. 
From the e-learning literature, several features were found to be associated with e-learning adoption; for example the 
need for learning to be a part of general organizational practice; positive attitudes toward using technology for 
learning at all levels within the organization; and an accessible and quality system. These features reoccurred as 
favoring easier implementation and acceptance of e-learning. Therefore questions pertaining to these finding were 
included in the tool. The development process resulted in four survey instruments, each of which focused on 
different target groups within the organization: (A) leadership survey, (B) human resources survey, (C) learner 
analysis survey, and (D) technology survey. 

The leadership survey (A) has two target groups: top management and mid-level management. The basis 

for these targets was the acknowledgement that adoption of e-learning is a change and any change must be 

supported by leaders within the organization (Annansingh & Bright, 2010; Purnomo & Lee, 2013). The 

distinction in management was also important since mid —level managers would be able to provide more 

detailed information on every day running of the organization. 

The human resource survey (B) is meant to gain opinions from the individuals who are supposed to know 

where knowledge lies in the institution and the learning needs of the organization. 

The learner analysis survey (C) targets employees that would be presumed to actively use and interact with 

e-learning. The literature recognizes that several employee characteristics can contribute to e-learning 

success and chief among them is self-efficacy, (Womble, 2008). The survey looks to gather learner 

perceptions regarding various aspects of readiness including organizational support, learning culture, 

learner access and attitudes. 

The technology survey (D) targets the individuals who are responsible for facilitating the use and 

acquisition of computer technology within the organization. 

The final aspect of the tool serves to collate the information gathered from the four surveys into as succinct 
document that can be used for decision-making. It is in the form of a final checklist and recommendations (E). The 
process is outlined in figure 3. 


Organizational 
Surveys (A and B) 


Learner Survey (C) 


Technology Survey 
(D) 


pue 3si9ayD |eul4 


PS) 
O 
O 
Oo 
3 
3 
O 
S 
ios 
o 
as 
Oo 
S 
Nn 
Gs 


Figure 3. The process of using the e-learning analysis tool 


Source: James-Springer (2016) 
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Evaluation 


Formative evaluation was conducted throughout the development process to determine the quality, 
efficiency and effectiveness of the tool (Van den Akker, Branch, Gustafson & Plomp, 2012). This was done through 
use of two types of experts who looked at construct and content validity of the tool separately. This phase of the 
research study revealed both strengths and weaknesses of the tool. Feedback was consistent in terms of the tool as 
practical and usable, which would capture relevant information. However, suggestions were to decrease the 
technical nature of the questions, and tailor some of the surveys for each target audience. This included carefully 
considering the self-efficacy of the information technology personnel and adding questions that would capture the 
idea of the personal learning culture of the individuals taking the surveys. These changes were taken into 
consideration in the final iteration of the tool. 


Findings 


The findings from this design and development study showed that it was difficult to separate this 
developmental research into the distinct categories of model or tool research. Tool research is context specific while 
model research is general (Richey & Klien, 2007), yet, this research resulted in a tool which is generalizable to 
various contexts. The deviation form tool design research also included a move from using case studies, as 
suggested by Richey and Klien (2007), and focused on gathering information from various contexts within the e- 
learning literature. This review of the literature resulted in a more generalizable final tool. 

In the absence of clear guidelines for tool research in Richey amd Klien (2007), it was necessary to look 
beyond IDT for guidelines in tool design. This procedure was informed by other disciplines. The methodology for 
this research drew on the works of Ellis and Levy (2010) and Nunamker et al. (1990) which featured information 
systems design approaches to tool development. Thus, the resulting methodology was an amalgamation of methods 
from IDT and computer systems design (Ellis & Levy, 2010). 

Design and development research allows for flexibility of procedure (Van den Akker et al. 2010; Richey & 
Klein, 2007). At any time during this design and development, there was a focus on more than one variable. Early 
on in the research process, it was recognized that separation of the variables would have been difficult due to the 
interrelated nature of the factors affecting e-learning. Several themes were identified in the literature, these themes 
were related in many areas. For example, learning organizational culture is an aspect which affects every individual 
in an organization, whether it is creating the environment for learning or benefiting from this environment. The 
learning culture also affects the processes used in supporting learning within the organization. The idea of learning 
culture is interrelated with organizational readiness, learner readiness and technology readiness. 

The artifact created in this D&D research is specifically designed to collect data within organizations from 
multiple perspectives. Much of the literature represented views of individual groups within organizations on the 
readiness of the organization; for example, Annansingh and Bright (2010) interviewed leaders within an 
organization and their perspectives and assumptions were used to determine e-leaning readiness. The limitations of 
this approach resulted in the idea of targeting several groupings within an organization. Additional findings of the 
literature review were that that the existing tools did not have an efficient way of collating data. Survey E serves this 
function and provides an efficient and succinct way of presenting information in the tool designed. 


Conclusions 


The artifact created looks at e-learning readiness in a systematic and interrelated way. The end product 
produced a tool intended to capture the content necessary for making decisions on e-learning readiness while at the 
same time being practical in application and use. Organizations show unique characteristics, and therefore, 
developing a general tool to determine e-learning readiness posed some challenges. Questions had to be carefully 
selected so that they can apply to most settings. There was also difficulty in keeping the design and the development 
process separate at some points in the research, since many decisions overlapped and influenced each other. The 
process used to come to the final product resulted in the amalgamation of various methodologies for tool 
development, these consisted of both IDT processes as well as Information systems processes. 

Design and development research speaks to inclusion of several variables at once (Van den Akker et al., 
2012; Richey & Klien, 2007) and this research was no exception. The design of the tool itself reflects the 
interrelatedness of the variables and should help influence e-learning adoption decisions. Overall the tools should 
collect data from various sources within and organization in order to give a comprehensive view of the degree of 
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readiness within the organization. As much as possible it was designed to be easily used to collate and compare 
information gathered. 
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Introduction 


Despite the projected future demand of Computer Science related jobs in the US, there are not enough 
students choosing to major in this field (Bureau of Labor Statistics, 2014; NCLS, 2012). This issue is particularly 
evident with underrepresented minority (URM) students: among 22,273 high school students who took the 
Advanced Placement (AP) computer science test, 4% were African American and 9% were Hispanic Latino 
(College Board, 2013). URM students are significantly less likely to persevere in Science, Technology, Engineering 
and Math (STEM) majors (Chang, Sharkness Hurtado & Newman, 2014). 

Biggers, Brauer, and Yilmaz (2008) identify “the perception of asocial community with low levels of human 
interaction including student-faculty and peer-peer’ as one of the reasons students decide to leave their CS major (p. 
406). A way to potentially improve the persistence of URM students in these majors is increasing the likelihood of 
students to engage in academic experiences, such as studying frequently with others (Chang, Sharkness Hurtado & 
Newman, 2014). 


Peer-to-Peer Interaction and Learning 


There are several instructional strategies aimed at promoting peer-to-peer interaction as part of classroom 
activities in STEM courses such as Peer Instruction (Mazur, 2017), Team-Based Learning (Michaelsen & Sweet, 
2008) and Pair Programming, which is mainly intended for Computer Science related courses (Williams & Kessler, 
2002). These strategies are based on Social Constructivism, a learning theory that suggests that knowledge is 
constructed through interaction with others (Vygotsky, 1978). 

Peer-Led Team Learning (PLTL) is an instructional strategy that is not conducted as part of the classroom 
activities, as the aforementioned strategies, but as separate sessions that are lead by student mentors (called Peer 
Leaders). The instructor is still heavily involved in the design of the sessions (called workshops) but is not 
physically present in them. 

PLTL was developed by Gosser et al. (2001) in the early 1990s to support students in Chemistry courses, 
however, it has been successfully implemented in several other areas, such as Biology (Batz, 2014), Information and 
Communications Technology (Sheard et al., 2011), Mathematics (Hockings, DeAngelis, & Frey, 2008), and 
Computer Science (Horwitz et al., 2009; Huss-Lederman, Chinn, & Skrentny, 2008; Murphy, Powell, Parton, & 
Cannon, 2011; Stewart-Gardiner, 2010). 


Peer-Led Team Learning Characteristics 


Gosser et al. (2001) identify the following characteristics: PLTL consists of a weekly workshop, which 
involves a peer leader and a group of six to eight students. The course instructors are closely involved with the 
workshops by holding weekly sessions with the peer leaders and developing challenging materials that are integrated 
with the course content. 

The peer leaders are recruited from students who completed the course successfully. They are trained in 
leadership skills and teaching and learning strategies. Their role is more as facilitators and not as teaching assistants 
or mentors. Some of the peer leaders’ tasks involve: 

clarify the purpose and goal of each workshop 
ensure full participation of the students 

create special opportunities for students 

build students’ commitment and self-confidence 
practice listening skills and questioning techniques 
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strengthen students’ skills and problem solving approaches 
reflect continuously upon their process for leading the workshops 


Our First PLTL Implementation 


Encouraged by several studies highlighting the implementation of PLTL as a potential instructional strategy 
to reduce attrition for URM students (Horwitz et al., 2009, Stewart-Gardiner, 2010) and also for female students 
(Murphy, Powell, Parton, & Cannon, 2011), we decided to implement PLTL in CST231 Problem Solving and 
Programming in fall 2016. Our main research questions about the implementation of the PLTL program were: 

1. What are the effects of PLTL on retention rates and grades? 

2. What is the students’ perception of PLTL? 

3. In what ways could the PLTL program be improved? 

CST231 is the first programming course that freshman students take as part of the undergraduate program 
in Computer Science. The class size is about 35 on average, and about 35% of them are URM students. As part of 
this course, students are able to edit, run, debug, and document computer programs written in C++. The course 
introduces basic programming skills such as conditional statements, logical expressions, loop structures, arrays, data 
files, and an introduction to object-oriented programming. 

Following Gosser et al.’s (2001) recommendations to implement the PLTL program, we recruited students 
who successfully completed the course the previous semester as peer leaders; we trained them in leadership skills 
and teaching and learning strategies; two-hour workshops were offered every week; instructors were closely 
involved with the workshops, holding weekly sessions with the peer leaders and developing challenging materials 
that were integrated with the course content. 

There were three sections of CST231 Problem Solving and Programming in fall 2016. An instructor taught 
two and a second instructor taught the third section. The weekly content was exactly the same across all sections. By 
the end of the previous semester, we had identified and recruited four students who had completed the course 
successfully and were willing to be Peer Leaders. Two of them had been tutors in previous semesters. We organized 
a two-hour training session in which we trained them to be Peer Leaders. They learned how to use several 
instructional strategies to facilitate the workshops, including peer problem solving and round robin techniques 
(Gosser et al., 2001). 

The workshops were initially offered three days a week, Monday, Tuesday, and Friday. Each workshop 
lasted about 2 hours and the same content was covered in the three weekly workshops. In this way, students could 
attend a workshop on the day that was most convenient to them. Workshop attendance was not compulsory but 
strongly encouraged. Each weekly course assignment consisted of five questions/problems, two of which were ill- 
defined problems that were addressed as part of the PLTL workshop. Consequently, students attending the 
workshops would then benefit by working on their homework assignment under the guidance of the peer leaders. 

The emphasis of the workshops was to focus on the problem solving process. Students worked in groups of 
four to six, depending on attendance. It was decided that computers shouldn’t be used during the sessions for several 
reasons: a) promoting peer-to-peer interaction by preventing students from working individually in a computer, b) 
preventing students from looking for answers to the problems presented, and c) forcing students to memorize and 
recall the syntax of programming in C++. 

Unfortunately, student attendance was a major issue: out of the 108 total students in the three sections, only 
about 50 students (46.29%) attended at least one workshop during the semester. Only two weeks had over 20 
students attending the weekly workshops but on average, there were only about 10 students attending the workshops 
each week. 

Both instructors were adamant about not assigning extra points for attendance, fearing that students would 
only attend to earn the extra credit. However, we did implement several other strategies to increase student 
attendance, such as: 

1) Instructors increased the difficulty level of one of the ill-defined problems and encouraged students to 
attend the PLTL sessions to work on it. 

2) Asurvey was conducted by the end of the first month of classes to identify a better day and time that would 
work for more students. Based on the students’ feedback, the Monday workshop was moved to a different 
time on Fridays. 

3) PLTL leaders came to class to give a short presentation about the benefits of attending the PLTL sessions 
and to answer any questions or concerns that students had. 
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Low attendance to PLTL workshops seems to be a common issue (Gafney & Varma-Nelson, 2008). 
Stewart-Gardiner (2010) reports that for several semesters almost no students participated voluntarily in the 
workshops. Even with extra credit, just two students out of 80 attended the workshops regularly. 

We conducted a survey to analyze the students’ perception about the PLTL program and identify areas of 
improvement. In total, 67 students from the three sections submitted the survey and 14 of them indicated that they 
had not attended any workshop. Nine of the students who did not attend any workshop reported that they were 
expecting to get an A as their final grade, and the remaining five were expecting to get a grade of B. The most 
common reason for not attending any PLTL workshop was that students felt they did not need them because they 
were able to solve the assignment problems by themselves. 

The number of students that dropped, failed or withdrew the class was similar to previous semesters, 
between 25 and 30% per section. However, due to the low workshop attendance, it was not really possible to judge 
the effectiveness of the PLTL program. 


Revised PLTL Implementation 


In Spring 2017 the PLTL program was implemented for a second semester but attendance was made 
compulsory. Indeed, some PLTL programs at different universities require attendance and participation (Gafney & 
Varma-Nelson, 2008). At Columbia University, students taking CS1 can get one unit of research credit for their 
participation in the PLTL workshops, with two unexcused absences resulting in a failing grade (Murphy et al., 
2011). 

When the class was made available for registration, students were notified that they were required to attend 
three days per week during the semester, and that the third day would be for a lab session (PLTL). Attendance was 
made mandatory, with four unexcused absences resulting in a failing grade. 

There were three sections of the class, each with about 32 students; each section attended a PLTL 
workshop separately. There were three peer leaders attending each workshop. One of the instructors met on weekly 
basis with the peer leaders for around 30 minutes to discuss the problems to be covered during the following 
workshop. 

Other than making the attendance required, everything else remained the same: students were grouped 
randomly in groups of six to seven; two ill-defined problems were assigned as part of the class and students worked 
on them in the workshops; no computers were allowed. After identifying a potential solution to a problem, one of 
the students in a group would write it down in a whiteboard for everybody else to see it and provide comments and 
suggestions. 

In spite of the required attendance and the joined efforts of the instructors and peer leaders, by the end of 
the semester, the percentage of students who failed, dropped, or withdrew from the class was still similar to previous 
semesters (25% to 30% per section). These results were somehow similar as those reported by Merkel and Brania 
(2015), who implemented a PLTL program for a Calculus I class but did not observe any gains in students’ learning 
or retention. 

We did not conduct a survey about the perception of students towards the required PLTL workshops. 
However, based on the opinion and observations from some of the peer leaders, the ill-defined problems used as part 
of the workshops might have been problematic for two reasons: 1) the problems might have been too advanced and 
might have tended to just benefit stronger students, and not those students who were still struggling with conceptual 
understanding, and 2) some of the problems were too difficult to be broken in smaller parts for analysis and to 
promote collaborative learning, as suggested by the PLTL literature (Gosser et al., 2001). 


Conclusion 


We implemented a PLTL program during two consecutive semesters with the goal to increase students’ 
performance in programming and problem solving, as well as to decrease the number of students who drop, 
withdraw, or fail from the Programming and Problems Solving class. Even though there was no significant 
differences in the percentage of students failing or withdrawing the class, it doesn’t necessarily means that a PLTL 
program is not effective. 

Given that PLTL participation was voluntary during the first semester, only very few students actually 
attended the workshops. The lesson we learned was to either apply more active recruitment or to make participation 
mandatory. During the second semester implementing PLTL, we learned that it is important to use the right kind of 
problems as part of the workshops (problems with the right level of complexity that can be broken into smaller sub- 
problems). 
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As part of our analysis, we should have also considered other factors that might influence students’ 
decision to drop or withdraw the class: CST231 is the first class in the Computer Science program and it is possible 
that some freshmen students might still be undecided about their career goals. Perhaps the PLTL program might be 
more helpful in sophomore or junior years, in which more students are committed to pursuing a career in Computer 
Science. 
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Index Descriptors: Distance Learning, Nontraditional Students 


Nontraditional students are multi-taskers. They work, they go to school, and they have family 
responsibilities. Sometimes they are successful with balancing their responsibilities. However, in many cases 
nontraditional students have little success. As Shapiro et al. (2015) reports, college cohorts who started in 2009 had 
graduation rates of 39.20% for students above the age of 24, compared to 58.6% for those who started their program 
under the age of 24. For online students, Patterson and McFadden (2009) drew a close correlation between being an 
older online student and student attrition. As a result, universities face two nontraditional student persistence 
impactors: nontraditional student external stressors and the intrinsic motivation needed to be a successful online 
nontraditional student (McClain-Smith, 2017). These impactors are areas that university’s do not have control over 
and can provide little help...or perhaps they can? 

While matriculating, online nontraditional students raise their children, deal with family crises, handle 
work duties and deadlines, while managing assignments, readings, and other learning activities within their courses. 
When external stressors become too much, nontraditional students are forced to prioritize their attention amongst 
family, work, and school. In turn, they face difficult choices. If stressors within college courses compete with 
stressors of life, students may choose to remove the easiest of all stressors, school. However, if instructional actions 
and course design reduce course stressors, students may be inclined to continue their courses and in turn complete 
their degree program. 

One way to address these issues is by providing online instruction with respect of the nontraditional 
student’s time. Student support services, along with course relevancy are often in the forefront of university online 
strategies. However, actual consideration of how course instruction and design affects student time constraints is an 
area in need of further exploration. Although instructors may allow some individual flexibility, in many cases course 
structure and learning activities may cause conflicts with nontraditional students' time constraints. 

There are strategies available, in the area of respecting the time of online nontraditional college students, 
without reducing quality or instructional rigor. Key constituents such as college faculty, administrators, and 
instructional designers, have the ability to implement these strategies. As leaders within their universities, courses, 
and course design, these professional educators are the change agents needed when nontraditional students take 
online college courses. During phenomenological research data collection, the researcher interviewed online 
nontraditional students about their experiences within their online courses and the implementation of autonomy 
supportive actions by instructors or through course design. Autonomy supportive actions for the online classroom 
include (a) providing choices where students may select assignments based on preference, (b) providing rationale of 
the importance of the course, content, assignments, and (c) providing opportunities for personalization by allowing 
flexibility (Lee, Pate, & Cozart, 2015). Study results reflect that through the aforementioned actions, time, support, 
and relevancy are themes that reoccurred within participant statements. The Time theme had the most significant 
number of participant statements, accounting for 48%. The Support theme came in second with 39%, while the 
Personal Relevance theme had 13% of identified personal statements (McClain-Smith, 2017). As a result, time and 
the respect of time in particular, significantly influences students’ ability to be intrinsically motivated to persist. 

Of course, there are variants of online delivery across universities, with some courses being synchronous, 
asynchronous or blended. However, within various delivery methods there are course design and instructor delivery 
practices that negatively affect the matriculation of nontraditional students. Below are shared samples of participant 
statements (primary resources), reflecting nontraditional student thoughts and experiences while taking online 
courses. In some cases, students faced online courses that demand lengthy or excessive amounts of synchronous 
sessions. During participant interviews, when asked about synchronous lectures Participant 2 stated the following: 

It can already be boring to sit through a lecture, but to sit through a lecture at your own computer with a lot 

of distractions at a time that might not be ideal to you can be very frustrating. I think I didn’t gain a lot 

from those experiences. It was frustrating with the timing. I was one of the few people on a different time 
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zone. So that made it even more difficult because the timings of the sessions were based on the university. 

For most people that worked out well, but for me it did not work out well. Sometimes it felt like wasted 

time. It was frustrating. Part of the reason for choosing an online program is because you want to be able to 

get things done at a time that’s good for you (McClain-Smith, 2017). 

An additional practice experienced by study participants was the restriction of viewable course content. 

This practice limits the nontraditional online student from moving ahead as needed based on personal circumstances. 
When asked about course flexibility and the availability of content, Participant 27 stated the following: 

I would have liked him to [put learning activities up all at once] because if I was off on the weekend I was 

working on schoolwork. I would have liked to see everything up front so I could work down the list. I don’t 

know why he didn’t post it. I requested it, but I don’t know why everything wasn’t posted at one time. 

Different professors are different. Some will post it all and some won’t. In the back of my mind, I was a 

little worried, because I didn’t know when I would find the time to do them (McClain-Smith, 2017). 

Within the same flexibility topic, when asked about course structure, Participant 2 stated the following: 

“Three weeks in there’s going to be a project and then a paper and then a big project. If I knew that in the 

beginning, regardless of if I was able to start on these things it gave me the ability to plan out my time and 

know if I needed to start something early. I’m going to start on this project weeks before it’s due and I will 
start on another at another time. Because of the fact that they planned it out so well with a clear structure, 
they made it very well organized for us. Some even provided the rubrics at the beginning of the course, 

even if we couldn’t complete it; it gave us the ability to plan our time” (McClain-Smith, 2017). 

Depending on the action, abating or implementing such actions may influence a nontraditional student’s 
time and subsequently impact the student’s intrinsic motivation to persist. Nevertheless, there are actions to 
implement within online courses to counteract students’ demotivation to persist. There are derived strategies that 
respect the time of nontraditional students based on perspectives of student’s struggles and successes experienced 
in online college courses. To help with the time crunch of nontraditional online students, these actions include 
providing structure, timing, and communications (See Figure 1). 


Practices to Help Non-Traditional College Students 
with Time and Persistence 


0e0 


Structure Timing Communications 


* Structured * Manageable + Clear expectations 


course content time frames and instructions 

and resources « Live lectures + Assignment rubrics 
* Viewable LMS that are 

content recorded 


©Tammy McClain-Smith, Ed.D. 2017 


Figure 1. Recommended instructor and course design actions to assist online nontraditional students with time. 


When examining course structure, well organized course layouts help eliminate confusion for the nontraditional 
student. Additionally, make learning management system (LMS) content viewable so that the nontraditional 
student can plan based on their time. In the area of timing, develop well-paced online courses, with learning 
activities provided within manageable timeframes. When using live lectures through a web conferencing tool, take 
into account lecture length and student access to lectures outside of live lecture timeframes. Recorded lectures are 
not only useful for nontraditional students, but also beneficial to traditional students. Finally, implement good 
communication by providing thorough instructions for learning activities. Additionally, rubrics are helpful in 
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communicating grading criteria. By using them, there will be no question of what level a student must perform to 
earn a particular grade. Most modern learning management systems provide the ability to embed rubrics within 
course assignment pages. Equally as important is providing the same rubric on assignment instructional documents 
for quick reference. The aforementioned actions are a sampling and considered as “low hanging fruit” that can be 
implemented easily within a course design or at a university policy level. 

Autonomy supportive efforts that respect the nontraditional student’s time can increase intrinsic motivation 
within online courses. Hence, better management of student external stressors result in the reduction of course 
stressors. Actions are easily implemented and can assist in providing further support for the nontraditional student; 
whether actions are in an individual course or college-wide. These recommendations may lead towards the support 
of autonomy of nontraditional students. While continual research will drill down the exact impact of each action, the 
recommended actions are a direct result of student reactions and recommendations. As a result, the implementation 
of these actions could possibly lead to higher student course and program persistence. 
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Abstract 


Say goodbye to working in isolation and hello to technology integration through innovative learning 
communities, both online and face-to-face. This presentation described and reflected on the organic and innovative 
processes that evolved during the creation and formative growth of an online professional learning community 
(PLC) in the context of higher education; the TILC (Technology Integration Learning Community). The TILC 
emerged and thrived in the context of budgetary constraints that limited the number of face-to-face faculty retreats 
and funds for professional development and conference attendance. 

The goals of our TILC are to promote change, foster collaboration, and lead the way for integrating 
technology among a group of faculty. 


Introduction 


In the context of institutional budgetary constraints that limited formal professional development 
opportunities, the Technology Integration Learning Community (TILC) emerged to link geographically dispersed 
faculty with the purpose of integrating technology to engage online students. This presentation traced the 
development and evolution of the TILC, described its achievements, and shared lessons learned. Presenters offered 
strategies to foster success and promote transfer in the online learning environment. 


Leading Educational Change 


This self-initiated and independently coordinated learning community critically adapted the main 
foundations of theory crafted by experts in PLCs (see Dufour, Dufour, & Eaker, 2008; Easton, 2011; Martin-Kniep, 
2004) and the foundations of andragogy (see Knowles, 1990; Linderman, 1961), to become a forum for professional 
development, technology integration and collaborative reflection to promote change in teaching and learning. 
Definitions of Professional Learning communities agree on PLCs as teams of educators who meet regularly to 
collaborate, learn, and improve teaching and learning. PLCs adopt a culture of collaboration that promotes data 
analysis, risk taking, and reflection. As a model of professional development, PLCs meet the qualities of effective 
professional development (Fogarty and Pete, 2007): they are job embedded, sustained, collegial, results-oriented, 
and practical. As a consequence, the work of PLCs can have real impact on students learning and on the institutions 
where they work. Because the context of the TILC is a College of Education in which most of the courses are 
teaching online, this PLC developed online. Its work is ongoing and it has been evolved for years in a collegial 
manner, based on the needs for collegial interaction, camaraderie, and professional development. The desired results 
that this learning community focus on are the improvement of learning through members and students’ engagement. 
The members of the TILC reflect on the concepts and theories that support our practice. However, what we learn 
during our meetings is practical and is transferred into our courses. 

Based on Knowles’ findings of andragogy or adult learning theory (Knowles, 1973), the TILC offers its 
members control on their learning, immediate utility of the knowledge and skills learned, a focus on issues that 
interest us, opportunities to anticipate and discuss the use of what is learned, expectations to improve faculty 
members’ performance as instructors and as designers, the ability to maximize the access to resources, opportunities 
to collaborate in a respectful and informal environment, and access to information that is appropriately paced. 
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Members of the PLC control the content addressed during the meetings by proposing and selecting what they want 
to learn and how they want to do it. Usually, one of the members of the PLC gives a brief presentation on a learning 
tool or strategy and leads a workshop for everyone to learn how to use it. In addition, the team discusses strategies 
for immediate integration. Every term faculty members share their experiences implementing one of the tools or 
strategies learned during the previous term. 

Change for this team involves promoting engagement through integration of newer technologies that foster 
an environment of innovation and productivity for the students in our courses. Members of the TILC lead change by 
teaching each other how to use innovative tools and technologies, and discussing how to integrate these tools and 
technologies into our courses. Change is analyzed to evaluate our own learning and progression within our learning 
community. 


History of our PLC 


Our TILC emerged as a result of a Blue Sky focus group to brainstorm how to enhance student engagement 
at the Abraham S. Fischler College of Education (FCE) at Nova Southeastern University (NSU). We decided to pilot 
an iPad initiative with full support from administration, with the goal of integrating more technology into our 
courses. In 2012, we piloted the initiative with a small Master’s in Athletic Administration program, taught the 
program faculty how to use the iPads, recommended a number of helpful apps, and challenged them to integrate 
mobile learning into their curriculum. Upon completion of the pilot, we evaluated our efforts to determine what 
worked (1.e., collaboration, administrative support) and what did not work (i.e., lack of accountability, limited 
training; Lacey, Gunter, & Reeves, 2014). 

The need for additional training evolved into the development of an online learning community of eight 
NSU faculty members to promote technology integration during the Fall of 2013. At first our learning community 
focused on mobile learning, where each faculty member was responsible for learning about and teaching the rest of 
the learning community about certain mobile learning applications. We posted app reviews on a Wiki, but instead of 
just reviewing the app, we went a step beyond to discuss how we could integrate them into our courses. It was the 
integration ideas that really made our learning community unique. During the next couple of years, we evolved into 
a true technology integration learning community based on a collaborative training approach, where everyone was 
responsible for their own learning. We meet monthly to learn about new tools or technologies and always end the 
training with a discussion of how to integrate our newfound knowledge into our courses. To hold each other 
accountable we challenge each member of the TILC to integrate at least one new tool/technology into their courses 
each semester. 

To guide our trainings, we developed the TILC Instructional Framework incorporating the social, 
collaborative, and evaluative components of our learning community (Please see Reeves et al., 2015 for a discussion 
on the framework). 
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Figure 1. TILC Instructional Framework 


Best Practices 


Best practices for our TILC include defining clear and common goals, using a common communication platform, 
leading change by teaching each other, promoting transfer by discussing integration ideas, and holding each other 
accountable by challenging each member to integrate one new tool or technique into our courses each semester. 

The goals of our TILC are to increase student and faculty engagement and promote integration of 
technology into our courses. We use a common video communication platform so that every member of the learning 
community feels a sense of connection. Although the members of our TILC are geographically dispersed, we do not 
feel as if we are working in isolation. Activities and training sessions are developed as a collaborative group, rather 
than thrust upon the community by the administration. This sense of community and collaboration is a key pillar to 
our learning community’s effectiveness. 

We hold monthly meetings where one member of our TILC leads change by teaching the other members 
about a new innovative tool and technique. Giving members the opportunity to teach each other creates a sense of 
engagement, responsibility, and accountability that benefits the community. In addition, offering the choice to teach 
or discuss a specific tool addresses two of the characteristics of the adult learner: the need to (a) control our learning 
and (b) focus on issues that concern us (Knowles, 1990). At the end of each meeting we promote transfer by 
discussing how to integrate these ideas into our classes. Regardless of skill level, group members help each other in 
an authentic way to move the group forward as a social community. Finally, we hold each other accountable by 
challenging each member to integrate one new tool or technique into our courses each semester. 

The members of our TILC share, implement, and reflect on user-friendly and free technology tools that are 
easy to access not only for us, but also for our students. This is fundamental to our learning community: keeping it 
simple and practical. The members of our TILC have employed leadership skills by coaching colleagues during 
faculty retreats; at meetings; and local, state, and international conferences. 

The presentation targeted K-12 educators and higher education professors and administrators who want to 
lead change by engaging students through the integration of technology. This presentation offered recommendations 
to establish a successful learning community related to any content area or interest. Modeling the ways in which this 
learning community interacts, we had traditional presenters (in the assigned conference location) and presenters at a 
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distance via video. This presentation reflected on best practices for design, recruitment, and sustainability of a 
learning community. We also shared with the audience lessons learned, our instructional framework, and tips for 
promoting transfer in the online learning environment. 
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Abstract 


This roundtable shared best practices and tools for leading and learning to foster engagement. This 
presentation showcased how to develop a learning community that integrates technological tools and strategies to 
promote collaboration, communication, and personal growth. In the context of institutional budgetary constraints 
that limited formal professional development opportunities, the Technology Integration Learning Community 
(TILC) emerged to link geographically dispersed faculty with the purpose of integrating technology to engage online 
students. 


Introduction 


Our Online Motivational Group (OMG) shared and discussed with the audience their ideas for creating faculty and 
student engagement opportunities. In this presentation, our Technology Integration Learning Community (TILC) 
discussed our journey and presented a brief outline about some of the tools we use to increase engagement in our 
courses, including tools for communication, collaboration, and productivity. 

We provided participants with effective practices to promote collaboration and problem solving within a 
learning community and led a discussion on enhancing engagement in the classroom by discussing the technology 
tools, experiences, and outcomes from our TILC. By sharing the lessons we have learned, participants learned about 
our successes and roadblocks to implementing technology enriched learning environments to create engagement 
opportunities. 


Creating Faculty and Student Engagement Opportunities 


Many educators and learners are technologically competent using multiple digital and media devices and 
applications simultaneously. The connection between theory and application is bridged simply by sharing with 
colleagues, and reflecting about our own practices while referencing what the literature stated in a very real-life 
fashion. The goals of our TILC are to promote change, foster collaboration, and lead the way for integrating 
technology among a group of faculty. 


Tools and Strategies for Enhancing Engagement 


The following technological tools and strategies were briefly introduced to create engagement 
opportunities. These were organized under three main categories: communication, collaboration, and productivity. 

Various communication tools were introduced including some for communicating more effectively 
(Remind and Google Voice), sharing information freely through the use of Blogs (e.g., Blogger) and microblogs 
(e.g., Twitter), staying in touch with family and friends via social media (e.g., Facebook), and streaming live video 
(e.g., Periscope and Facebook). We briefly described each of these tools below. Remind is a website that provides 
users a way to text message or email students and parents “where they are.” This application divides courses into 
separate sections and can be utilized to send specific messages as needed to specific groups. Students and parents 
join the class using a code. Neither teachers nor student see other users’ phone number. Google Voice is a free 
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communication tool that allows communication using all of your electronic resources including: telephones, mobile 
devices and a computer. This allows the users to gain access to needed information using any tool that is available 
at the time. Blogs and Microblogs are online diaries of thoughts published by an individual or a company. Blogs 
and microblogs have many educational applications and can be used to foster discussion in lieu of a traditional 
discussion forum. Blogger and Twitter seem to be the standard or most utilized blog and microblog; however, there 
are numerous social media blogs available to users from a variety of website or application. Acquiring good 
information quickly and easily seems to be everyone's goal today, and these tools respond to this perceived need. 

Keeping in close contact with today’s learners, colleagues, or stakeholders is paramount and simple with a 
variety of asynchronous or synchronous social media websites. Options with this tool include public and private 
websites. In emergency situations, social media is already the first communication method utilized to notify people 
of event cancellations, changes in plans, or simply an update about any topic. We provided examples of how to 
integrate social media into our courses, including live streaming, which enables you to transmit live video and audio 
coverage of any event over the Internet. Various software application such as Facebook or Periscope make this 
activity easy and simple to complete. These can be used for courses where an activity must be documented or for 
spontaneous instruction from the classroom, home, or relevant locations. These tools can provide a level of interest 
to a lecture that might otherwise lack quality interaction. 

Collaborating in today’s digital world is essential. Google products lead the way for collaboration. Google 
docs allows users to work together to complete activities online in real-time creating a true collaboration experience 
with other users. Other Google products were introduced, as well various collaboration tools from the roundtable 
members. Google slides was a favorite tool to create presentations collaboratively, Google forms allows colleagues 
to create together surveys, quizzes, and questionnaires. Google sheets is the Google collaborative version of Excel. 

Finally, productivity tools are at the top of any educator’s list. One of our favorites is Pinterest, which is an 
online repository for storing information. We shared a number of great and innovative ways to integrate Pinterest 
into the curriculum to enhance engagement. 

Another must have in any educator’s toolbox is Flipped Learning. A Flipped Classroom is a model of 
blended learning in which students learn content usually at home, and work is done in class with teachers and 
students discussing questions and solving problems applying the knowledge they acquired at home. There are a 
variety of tools and applications that enable this process to flow smoothly and effectively. Video Multimedia Tools 
are paramount to the flipped classroom. Leading the pack is YouTube which is a video sharing service that allows 
people to watch videos posted by other people and upload videos of their own. Students whose professors embedded 
YouTube links reported the “videos were exceptionally helpful” in increasing their understanding of the course 
material, allowing them “to grasp an understanding of many issues that were abstract or complicated.” 

In conclusion, this presentation targeted K-12 educators and higher education professors and administrators 
who want to create faculty and student engagement opportunities. The discussion highlighted various tools and 
applications used to engage students and colleagues. The members of our TILC reflected on best practices for 
evaluating, integrating, and sharing technology tools to enhance engagement in teaching and learning. We shared 
lessons learned from integration and briefly showcased various tools and strategies that had been used by our TILC 
and have been compiled in several products, including an online user manual, a social media blog, and a website. 
Finally, we were excited to learn that other educators in the US and abroad use similar tools to enhance engagement 
in their classrooms. 
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Abstract 


This session will introduce attendees to the Future Ready Schools (FRS) movement and specifically the 
librarian’s role within the movement. Frequently serving as the technology leader within the school, the librarian 
will play a key role in leading learning for change. In a FRS, students and teachers will need access not only to 
quality digital tools and resources but also to a certified librarian who can locate, evaluate, and curate Open 
Educational Resources (OERs). 


Leading from the Library 


Librarianship no longer looks like card catalogs, dusty stacks, checkout slips, library cards, and overdues. Today’s 
librarians teach digital citizenship skills and how to differentiate between real news and fake news, provide 
professional development sessions for teachers, teach technology skills for students, help students and teachers find 
and evaluate digital resources, help students hone their MakerSpace skills, provide technology classes for parents, 
teach coding, manage the school’s social media and online presence, and maintain a digital and print collection 
along with all the other tasks of traditional librarianship. 
The landscape of most libraries is changing. Today, it is not enough simply to circulate books and offer a reading 
incentive program. In a time when funding for libraries continuously sits on the chopping block, librarians have had 
to redefine their role within the learning community. This is why the Future Ready Schools (FRS) initiative is so 
important. 

The FRS initiative calls librarians to step into a strong position of leadership. In looking at the layout of the 
FRS framework, librarians can clearly lead learning for change. The FRS framework is illustrated in Figure | as a 
wheel with eight pieces defining the role of a Future Ready Librarian (FRL). Of the eight sections of the graphic, 
three of them support the librarian as leader in a learning community: Collaborative Leadership; Curriculum, 
Instruction, and Assessment; and Personalized Professional Development. The FRS initiative defines tasks for each 
of those eight sections (Figure 2). Collaborative Leadership calls for the FRL to lead beyond the library reaching out 
to teachers to identify opportunities to co-teach and plan. Personalized Professional Learning is best provided by the 
librarian who already knows the school’s faculty, the school’s technology and infrastructure, and the school’s goals 
and objectives for improvement. The Curriculum, Instruction, and Assessment section places the librarian in a 
powerful instructional support role through instructional partnerships, empowerment, and curating resources. But 
how best can librarians lead beyond the library? The answer: Open Educational Resources (OER). 
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What Are Open Educational Resources? 


OER are those resources that afford users specific rights: reuse, redistribute, revise, and remix. The term 
Open Educational Resources was first used at the 2002 conference of the United Nations Educational, Scientific, 
and Cultural Organization (UNESCO). UNESCO defined an OER as “teaching, learning and research materials in 
any medium, digital or otherwise, that reside in the public domain or have been released under an open license that 
permits no-cost access, use, adaptation and redistribution by others with no or limited restrictions” (“Open 
educational resources,” 2017). The Four R’s of Openness establish the principles that must include the ability to 
reuse, revise, remix, and redistribute with copyright licensed by Creative Commons (Hilton, Johnson, Stein, & 
Wiley, 2010). Although there are several licensing options available through Creative Commons, creators can 
choose Attribution Only to facilitate the highest levels of openness, allowing users to reuse, revise, remix, and 
redistribute as needed. 

The mindset of sharing and generosity coupled with the ability to customize the resource and personalize 
learning for the students are among the greatest benefits of OER integration. “Education is sharing. Education is 
about being open” (Wiley, 2010). 


Why Open Educational Resources? 


OER integration has really begun to take off among college professors seeking to do their part to make higher 
education more accessible and more affordable. However, with ever-shrinking budgets, it is difficult for public 
school districts to justify purchasing new textbooks for every student on an adequate replacement schedule. For 
instance, in the library, guidelines suggest weeding the collection and discarding books more than five years old and 
in some cases such as computer science, every three years. It seems excessive, but students are done a great 
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disservice when provided with access to out-of-date information. Many librarians live by the mantra that “no 
information is better than misinformation” and connecting students with digital information sources can be the best 
solution for meeting the information needs of patrons. 

Personalized learning is another benefit of OER usage. A survey of 600 K-12 teachers who integrate OER 
in their classes indicated their preference for OER simply in the ability to adapt resources (de los Arcos, Farrow, 
Pitt, Weller, & McAndrew, 2016). Picture a lesson on meteorology. In a classroom using a print, copyright- 
restricted textbook, the teacher is locked into the textbook’s examples. In an OER classroom, that same unit on 
meteorology can become more meaningful when the open data collected through the school’s WeatherSTEM unit is 
pulled into the scenarios presented to the students. Picture analyzing sentence structure where the sentences feature 
scenarios and people that are familiar to students. Solving math problems that are connected to current community 
issues makes learning real. OER and the ease with which they can be adapted make personalized learning possible 
and easier than ever because truly open resources enable the user to easily edit the content of the resource without 
the need for additional software or equipment. 

Another benefit is in the ability to guide students in evaluating sources. There is great value in students 
seeing that information sources are found in other locations besides textbooks. Today’s students are “digital 
natives,” having never lived in a time without Internet (Prensky, 2001). Most learned to use a tablet efficiently 
before they learned to use a fork. By incorporating digital resources, the focus of classroom instruction shifts away 
from the teacher as the center to a more student-focused approach. A classroom can easily take an inquiry-based 
approach where the digital resources help students answer questions on the fly and can often lead to greater depth of 
knowledge. 

As classrooms become more inquiry-based and student-focused, there are greater opportunities for 
collaboration among students. Students in Canada can collaborate with students in Georgia and Maryland via 
Google Hangout to compare each area’s native plants and animals. Students interested in similar issues can work 
together to solve those problems even when working together happens across the country. Collaboration can happen 
easily within the classroom and around the world through the use of OER and the mindset they bring. Students can 
network with other individuals within the school or with students from a school across the country to solve 
problems. Students can share their data from experiments and compare results with students anywhere. 
Collaboration and problem solving are prime skills for students to develop, and OER integration offers a wealth of 
opportunities to do so when they learn to think outside the textbook. 


How Can Librarians Support OER Integration? 


In considering “the ‘whole school’ view, the librarian is in a key position to contribute to the development 
of strong professional learning communities through professional development and technology integration” (Dees, 
Mayer, Morin, & Willis, 2010, p. 10). Many librarians already serve in leadership positions within the school, but 
there are specific roles they can play to better facilitate the integration of OER. In fact, when carefully examining the 
Future Ready Librarians infographic, there are a number of sections that tie directly to librarians leading learning for 
change specifically through OER. A Future Ready Librarian must see his/her role beyond just the four walls of the 
classroom to the walls surrounding the school community. Although many of the segments support the initiative, 
OER integration is addressed specifically in four segments of the infographic: 


I, Curriculum, Instruction & Assessment: Curates Digital Resources and Tools 
2: Curriculum, Instruction & Assessment: Builds Instructional Partnerships 

3. Personalized Professional Learning: Facilitates Professional Learning 

4. Collaborative Leadership: Leads Beyond the Library 


Curriculum, Instruction & Assessment: Curates Digital Resources and Tools 


Librarians have always been curators. Traditionally they have been curators of print resources: fiction and 
nonfiction books, teacher resources, magazines, and newspapers. Today there is a need to curate digital resources 
and specifically OER in an effort to make these resources more accessible. A Future Ready Librarian “leads in the 
selection, integration, organization, and sharing of digital resources and tools to support transformational teaching 
and learning and develop the digital curation skills of others” (Future Ready Librarians, 2016). 

Currently Follett, provider of the library management platform Destiny, has built in pieces to curate OER 
through their service. So a school whose library management system is Destiny has the ability to search for and 
organize OER in addition to their books and digital resources. While Destiny provides access to OER included in the 
Destiny databases, Follett has recently released the latest updates to Destiny including Collections. This is basically 
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an online file cabinet where patrons can create collections of resources to include websites, YouTube videos, OER, 
print books within the library, digital books, and databases. For instance, teachers in a school who developed an 
interdisciplinary unit on the recent eclipse might have created a Collection called Eclipse. 
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In that Collection, a patron might find the print library resources on eclipses and all the websites collected 
in the creation of the Collection. Teachers and other collaborators can quickly add resources to the Collection by 
clicking the “Add to Collections” bookmarklet. 


FJ Oe) | (Lou Coty | Details Q search y was 


37 Items H Vv Filter | Showing 37 of 37 items Sort by: Most Recent ~ 


ils 
Heather Morin 4 

Use Your Smartphone to The Solar Eclipse In Varanasi - 
Eclipse Photograph the Solar Eclipse Wonders of the Solar System - 

‘Aftiole Series 1 Episode 1 Preview - 
MONIES CAMIPEETELE % http://www.skyandtelescope.com/astro. BBC Two 
: ‘ : ‘ % Video 

Grades If you can't resist taking pictures of https://www.youtube.com/watch?v=eOv. 


the solar eclipse with your 
smartphone - we know, it's so 
tempting! - read this first. 


More about this episode: 

http://www. bbe.co.uk/programmes/b 
O0qyxfb 
http://bbe.co.uk/solarsystem In this 
spell-binding series, Professor Brian 
Smartphone Solar __ pictures Cox visits some ... 


Subjects 


STEM/STEAM 


Eclipse § Photograph 


When An Eclipse Meant The 
End Of The World 


This service is also provided from other platforms and vendors. Some librarians have been able to use 
Google Drive to curate OER by setting up shared folders and team drives. CK-12 is an OER platform which 
provides Flexbooks, digital resources, videos, and simulations for classroom instruction. Flexbooks are online 
textbooks that are fully customizable by the individual teacher. The resources found in CK-12 are easily shared 
through a number of learning platforms such as Google Classroom, Canvas, and Edmodo (CK-12 Foundation, 
2017). Although there are a number of resources available through which OER can be curated, the librarian must be 
willing to be the classroom teacher’s instructional partner. 
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Curriculum, Instruction & Assessment: Builds Instructional Partnerships 


To become an effective instructional partner, the librarian must be visible and willing to reach out to 
teachers. Some librarians attend weekly professional learning meetings and curriculum planning meetings. Many are 
mindful to stay abreast of changes in curriculum and standards. By knowing the requirements of the curriculum, 
librarians can prepare collections of resources for teachers to build those instructional partnerships. Through events, 
functions, and programming planned through the library, teachers gain a better awareness of the purpose and role of 
the media center and specifically of the librarian. A member of the school council for the Chicago Public Schools 
recently penned a letter of support in the Chicago Tribune saying, “librarians are essential partners for teachers and 
mentors for students in the buildings where education takes place” and “cannot be replaced by classroom libraries, 
digital libraries, or even by their colleagues in public libraries who are increasingly asked to serve in their stead” 
(Walter, 2017, “A 21st Century Education,” para. 3). 

In the state of Georgia, a library media specialist’s professional certification is classified as service. A 
Future Ready Librarian provides a valuable service to the teaching community by locating, vetting, and curating 
OER, and also by empowering teachers to be OER users and creators through appropriate and relevant professional 
learning opportunities taught by the librarian. Servant leadership and having a strong desire to help others be 
successful plays a huge role in building instructional partnerships. 

Co-teaching is another opportunity to build instructional partnerships. The librarian might offer to teach 
topics such as digital literacy, technology application, resource evaluation, conducting research, and accessing 
resources. When pairing the librarian’s knowledge of information access with the classroom teacher’s content 
knowledge, powerful lessons can be developed and delivered, and a great extension is that the classroom teacher 
views the librarian as an instructional partner. 

Probably the most valuable service the librarian can provide is a listening ear. As teachers visit the library in search 
of resources for lessons or to check-out instructional DVDs or teacher support materials, this is a perfect opportunity 
to share about OER, to offer opportunities to co-teach, or to suggest working together to identify digital resources 
that meet the teacher’s specific need. 


Personalized Professional Learning: Facilitates Professional Learning 


Teachers frequently attend professional learning opportunities taught by individuals are not a part of the 
school community. A district purchases a new digital tool, and that company sends a representative to provide 
professional development. Many times, however, professional learning deals with resources not currently available 
at a school on devices to which teachers do not have access. Teachers need relevant and appropriate professional 
learning opportunities, and one of the best facilitators is the school’s own librarian. This is the person who is 
familiar with the devices on campus and how many there are, what the network infrastructure is, what programs are 
in use, but more importantly, the librarian is familiar with the people who will be attending the trainings. The 
librarian knows who teaches what subjects, how much teaching experience they have, and how comfortable they are 
with technology. This type of knowledge is built through instructional partnerships and will enable the librarian to 
work more effectively with groups or individual teachers. 


Collaborative Leadership: Leads Beyond the Library 


It would be easy for a librarian to sit at the circulation desk all day, checking in and out books, evaluating 
the library collection, oblivious to the teacher’s needs. However, if a librarian is to be a forward-thinking, Future 
Ready Librarian, he/she must take time to discover new and innovative teaching resources and strategies, maintain a 
strong professional learning network, hone old or learn new skills by attending professional learning, and 
continually self-reflect on the effectiveness of library programming. Building instructional partnerships, conducting 
professional learning, and curating resources requires the librarian to assume a leadership role. Volunteer to serve on 
leadership committees to give a voice to the library while raising awareness of OER. Advocate for students and 
teachers in the adoption of devices, software, and other resources to include OER. 


Conclusion 
A Future Ready Librarian must consider that librarianship means more than just curating and organizing 


print books. As more and more people turn first to digital resources, the librarian as the information professional is 
in the position to guide patrons in finding, evaluating, and using digital resources, including OER. Public school 
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budgets are continuing to shrink, and there is less possibility of adopting new textbooks on a regular schedule. 
Transitioning to OER in place of copyright-restricted, printed textbooks is going to require strong leadership 
because change can be difficult. Some teachers will have to be eased into the change; others will willingly jump feet 
first into the change, but a Future Ready Librarian will lead the way knowing that “the more open we are, the better 
education will be,” (Wiley, 2010). 
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Abstract 


The Duolingo English Test (DET) is an English proficiency exam that “generates a CEFR-aligned score” 
and compares to “other major assessments such as the TOEFL and the IELTS.” (Duolingo English Test, 2016). The 
test is designed to provide an English proficiency score and “...make standardized tests more accessible to Duolingo 
users, particularly those outside the U.S., who want to apply for jobs and schools that require English certificates.” 
(Rodriguez, 2014, p. 1). The test provides “a video that showcases the applicant’s practical communication skills” 
and “a rigorous, quantified score that signals the applicant’s English level.” (Duolingo English Test, 2016). Paired 
with the Duolingo English app, data the DET provides can help teachers and administrators benchmark English 
proficiency across their English as a Second Language population. For this evaluation, the evaluation team 
negotiated with Duolingo for temporary, free access to the test. Based on the evaluation of the product, we created a 
learning technology evaluation framework CPR: Context, Pilot, and Report. 


Introduction 


Duolingo is a technology company based out of Pittsburgh, PA that offers customers different products 
around language learning. Since its creation in 2012, the company has expanded to include three main products, 
Duolingo, Duolingo English Test (DET), and TinyCards. Duolingo, original product created by von Ahn and 
Hacker in 2012, is a free language learning platform offering customers a way to learn a variety of languages on the 
web or through a mobile application. DET, created in 2014, is an online proficiency testing that allows customers to 
showcase his/her level of knowledge with the English language. The DET English test, unlike the language learning 
tool, is not free to use. The funding model for this product is $49 a person, marketed heavily towards universities 
and businesses using it to accept non-native English speaking students or employees. 


About the DET 


A common issue for schools that work with English Language Learners (ELL) students is categorizing the 
learner’s English language proficiency. Currently, this is often evaluated by the Test of English as a Foreign 
Language (TOEFL) scores. However, the TOEFL is expensive. Prices vary by country, but usually fall between 
$160 and $250 USD. A primary outcome of the DET is to certify the test taker in the English language. A potential 
reason Duolingo decided to go in this direction is due to the cost of the TOEFL test as well as the time it takes to 
receive the results. Duolingo’s version allows users to take the test in the comfort of their own home via the web, 
cost significantly less, and allows test takers to post results to social media and institution applications. 

One concern would be whether DET would be an acceptable comparison the TOEFL scores. Ye (2014), 
found that amongst a sample of non-native English speakers, “scores from the Duolingo English Test were found to 
be substantially correlated with the TOEFL iBT total scores, and moderately correlated with the individual TOEFL 
iBT section scores, which present strong criterion-related evidence for validity” (p. 11). Despite it being proctored 
online, there are multiple safeguards in place such as having a live proctor watch a recording of the assessment. The 
test is said to provide universities with “a video that showcases the applicant’s practical communication skills” and 
“a rigorous, quantified score that signals the applicant’s English level.” (Duolingo English Test, 2016). These 
benefits make the DET a great option for English language testing. DET administration opens a world of 
possibilities for future applications. 


Evaluation Framework 


To analyze and evaluate the DET, we created an evaluation framework called CPR. CPR stands for 
Context, Pilot and Report. Often in learning technology, proper evaluation is costly and time consuming. By using 
CPR, administrators and stakeholders can focus on elements of an evaluation that can lead them to data informed 
decision making. 


Context 
For the first step, Context, we took a hard look at the fundamentals of DET. Using our log ins, each 
member of the team stepped through logging in and taking a practice test. We each took notes about our own 


experiences and meet to discuss the pros and cons. We then created several evaluative questions around our initial 
needs for the DET. From here, using our initial thoughts and guiding questions, we created a practical approach to 


205 


reviewing the product. This allowed us to critically explore the technological requirements, cost, and feasibility of 
mobile access. 


Pilot 


For the second step, Pilot, we identified a population to test out DET. Using the evaluative questions as 
guides and our prior experiences, we surveyed the population after their review of the product to gather additional 
evidence for the evaluation of DET. The pilot step is often the one that is over looked, but is critical to gather 
perspective from the users. 


Report 


Lastly, for the Report step, we gathered the data and began to create a story. We had the reporting step in 
mind during the context and pilot phases. We asked ourselves, what information do we need to answer the 
evaluative questions? The results of the population survey as well as the DET scores and other valuable information 
gathered from population were critical in making inferences about the product viability. 


Recommendations 


From the CPR process, recommendations for product improvement were shaped. While the DET is a well- 
rounded product, there are several recommendations that should be considered to make the product better. When 
evaluating the DET, our group approached it from multiple perspectives including the user experience, educational 
value, and future implications. Based on these lenses, we have identified four recommendations for the DET: 
enhance the certificate the user is awarded after the exam, create Section 508 of the American with Disabilities Act 
test items, provide instructions to the Duolingo English Exam in a user’s native language, and provide percentage 
breakdowns on the report of the test. 


Certificate Improvements 


Once a user completes DET and their exam has been reviewed by the Duolingo reviewer, the user will 
receive an email notifying them of their certificate. To access the certificate, a link is sent to the user. On this link, 
the certificate includes the following information: a photo of the user from the exam, the name of the user, the date 
the exam was taken, and the level assigned to the user (beginner, intermediate, expert, etc.). Along with the level, 
Duolingo provides context about each of the levels. One improvement that should be considered is that it seems the 
level information pre-populates in a section with a character limit. In the example of the certificate above, the 
description of being an expert in English cuts off. The user and anyone with the link to the certificate is left without 
a full description of the level. Additionally, going more in depth on what a level means and sharing how it relates 
to the Duolingo app data would be helpful. Also, an improvement that could be considered is listing all of the levels 
a user can earn. One way of doing this is to create an artifact to accompany the user certificate. This way users, as 
well as organizations that the user may share the certificate with, will have more information about the levels. 
Furthermore, the artifact can provide organizations information about the DET as a whole. Another 
recommendation for the user certificate is providing a secure URL. The way the DET certificate is shared currently 
is through a URL. This URL has no safeguards so anyone with the URL can access a user’s certificate. Depending 
on how the user feels about this, it would be an improvement if the URL had a captive portal or some type of 
safeguard to protect the information. 


Create Accessible Test Items 


While the DET does a great job with offering a variety of types of questions, it could be improved by 
taking in Section 508 of the American with Disabilities Act into consideration to create test items that are accessible. 
Throughout the DET, there are various types of questions such as choosing the best word from a drop down menu, 
speaking about a topic, and selecting words from a list that are English words. While this variety makes the DET 
unique, it could be improved by considering different formats that are accessible. For the items that ask users to 
choose the best word from a drop down menu, Duolingo should consider enabling keyboard shortcuts and arrow key 
usage to make this type of question more accessible. Many users who use screen readers rely heavily on keyboard 
shortcuts in the same way that other users utilize a mouse. Furthermore, the speaking of topic items would be 
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difficult for users who are deaf. Offering an alternative question type such as an essay would be ideal. One way 
Duolingo could help users who may have disabilities is to ask in the beginning while setting up the test if they 
require an accommodation. Duolingo staff could then reach out to the user before the test or perhaps Duolingo 
would provide contact information for the user to contact Duolingo directly in order to assist the user. 


Provide Instructions to the DET in User’s Native Language 


During our evaluation of DET, a group member served as a pre-test proctor, sitting in the room to assist the 
user to get the test set up. While a primary purpose of this was to make sure the users were set up properly, another 
reason for the pre-test proctor is because the DET has all instructions in English. Depending on the English level of 
the user, this may be a hindrance. Due to the specific instructions such as having no one in the room as well as 
taking photos of government issued identification, it is imperative the user understand the process in order to 
complete the DET without fear of it not passing audit review. If DET offered instructions in multiple languages, it 
could guarantee that users understand the protocol and policies of the test. As discussed above, we believed the test 
would allow users to set up the test in their native language. But, it did not work during the learners’ try-out testing 
when students tried to select a language other than English. 


Provide Percentage Breakdowns on the Report of the Test 


Since the DET has many different types of items, it would be helpful for a user to receive a breakdown of 
the percentage correct in each different type of question. One item type asks the user to talk about a specific 
incident, such as a time they had to wait for something. Based on the information in our evaluation, we believe that 
the speaking part of the exam is not factored into the overall score in the DET. Having a breakdown of the item type 
would help both the test taker as well as an organization receiving the score to make an accurate judgement about 
someone’s English abilities. 


Conclusions 


The Duolingo English Test, when paired with the Duolingo English app, could provide ELL instructors 
multiple types of data for their students. Students who take the DET will be provided a digital artifact and encounter 
multiple question types to test their English proficiency. Before considering a full adoption, an evaluation is 
recommended using the framework Context, Pilot, and Reporting (CPR). 
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Abstract 


This study used the qualitative research methods to explore the enhancing higher education student’s cross— 
cultural competence in ubiquitous MOOC instructional design model. The participants were undergraduate students 
who have experienced in a ubiquitous MOOC instructional model. The researcher used a reliability technique in an 
interview procedure by having multiple coder in the step of intercoder agreement. The findings divided into three 
codes: ubiquitous MOOC learning, cognitive dissonance, and recommendations for ubiquitous MOOC. 


Introduction 


MOOCs, the massive online learning course, usually allow open enrollment leaners to access the worldwide 
communities with a diversity of cultural background and communication. MOOCs have remaining increased articles 
discussed empirical evidence from case studies especially in higher education (Liyanagunawardena, Adams, & 
Williams, 2013).In Thailand, e-learning was promoted by Thailand Cyber University (TCU) which start over ten 
years, more than 800 courses, 185,000 registered, and participated by 45 institution members (Na-songkhla, 
Thammetar, & Chen, 2015). OECD/UNESCO (2016) suggested that Thailand should train instructors how to use ICT 
and arouse them to integrate it into curriculum. In 2016, the first Thai MOOC Platform called Thai MOOC was 
established by TCU, then almost 2 million students registered (Thai MOOC, 2016). Likewise, some Thai universities 
also have created their own MOOC with no cost for anyone interested. For example, Chulalongkorn University had 
more than twelve courses created by seven faculties (Chula MOOC, 2017). Mahidol University also had MOOC called 
“MUx’” and there were thirty-eight courses about the medicine contents (MUx, 2017). Not only Thai language MOOC 
but also English MOOC, there was more than thirty-five course for international students as AsianUx Academy of 
Asian University (AsianUx, 2017). 

The number of international students additionally increased for higher education institutions (HEIs) in 
Thailand (Bureau of International Cooperation Strategy, 2010; Chunpen, 2013). Since 2014, there were more than 
10,000 international undergraduate students in Thailand (The International Trade Administration, 2017). The Office 
of the National Education Council (2005) in Thailand has stated that Thai people should understand, accept, and 
appreciate the cultures different both their own and others. Moreover, previous studies (Bennett, 1993; Nieto & Bode, 
2008; Rasmussen, 2013), cross-cultural competencies help educators develop mutual understanding, human 
relationships, and broaden their worldview. Furthermore, the cross-culture confliction occurred would occur while the 
cultural difference students was learning together even the MOOC in Thailand. Instructional designers and developers 
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should make use of cultural sensitivity to reducing cultural barriers so as to design international online learning 
(Alabdullaziz, 2015). A crucial factor cross-culture competent is attitude (Abbe, Gulick, & Herman, 2007; Gabrenya 
Jr, Griffith, Moukarzel, Pomerance, & Reid, 2012) could develop by changing the attitude change through Cognitive 
Dissonance Theory (Huffman, 2012). 

In order to enhance cross-cultural competency and undergraduate students’ opinions about ubiquitous MOOC 
for enhancing cross—cultural competence (Plangsorn, Na-Songkhla, & Luetkehans, 2016), the researchers developed 
the ubiquitous MOOC model (U-MOOC) based on cognitive dissonance theory as instructional design model for 
creating the MOOC in everyday life that learner can enhance knowledge everywhere every time and the contents are 
available view on PC, tablet, or smartphone. The evaluation of e-learning should investigate how course design and 
how learner learn by learner viewpoint (Khan, 2005). Although the U-MOOC could improve the undergraduate 
students’ higher cross—cultural competence, it would be necessary to remain developing the model. An Exploration 
of the Enhancing Student’s Cross—Cultural Competence in U-MOOC would be reflect the students’ feedback to 
improve the MOOC instructional model. 


Research Objectives 


The purpose of this study was to explore the enhancing higher education student’s cross—cultural competence 
in ubiquitous MOOC instructional model. 


Research Methods 


Qualitative research approach was employed to an exploration of the enhancing student’s cross—cultural 
competence in ubiquitous MOOC instructional model. The researcher conducted an informal unstructured interview and 
collected data from six participants. The participants were undergraduate students who had experienced learning in the 
MOOC designed by a ubiquitous MOOC instructional model. This number of participants was relevant to Polkinghorne 
(1989), who recommended that the interviewees should range from 5-25 individuals and have related experience with 
the phenomenon. The unstructured interview began with an open-ended question and built interviewees next questions 
based on what they said (Savenye, 2014). 

The researcher developed the interview protocol by focusing on the cognitive dissonance for enhancing higher 
education student’s cross—cultural competence in ubiquitous MOOC instructional model. The interview protocol was 
developed based on the instruction of ubiquitous MOOC following Creswell and Plano Clark (2011) who stated that the 
interview protocol compose with four parts of interview questions; (1) introduction question, (2) ice-breaking question, 
(3) sub-question, and (4) closing question. 

Interviews occurred at the end of the course which design by U-MOOC instructional model. The researcher 
contacted potential participants and asked them if they would be interested in participating. Each interview was 20-30 
minutes in length, and all the interviews were audio recorded and transcribed verbatim by the researcher. 

The researcher used a reliability technique in an interview procedure by having multiple coder in the step of 
intercoder agreement (American Psychological Association, 2010). All interview transcripts from the individual 
interviews were imported into MAXQDA 12 software. An inductive coding procedure was implemented as an analytical 
method in order to explore the enhancing student’s cross—cultural competence in ubiquitous MOOC instructional model. 


: : Procedures: Products: 
Develap antervien. - List the issue of the question - Interview protocol 
protocol - Consider interview question 
v 
Procedures: Products: 
Informal interview - N= 6 participants - Interview data 
- Informal interview 
i - Audio recorded and transcribed verbatim 
Procedures: Products: 
Intercoder agreement - Explain the coder about the research information - 3 themes 
and how to use MAXQDA software 
- Make a coder agreement 


- Inductive coding 


Figure I Research procedures 
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Research Results 


The results were divided into three themes: (1) U-MOOC learning, (2) cognitive dissonance, and (3) 


recommendations for U-MOOC, as follows. 


Theme 1: U-MOOC learning, this theme was explored how the respondents’ behavior in the MOOC that 


design by U-MOOC instructional design. The opinions on U-MOOC learning could divided into eight dimensions: 
(1) learning device, (2) learning motivation, (3) learning tools, (4) e-learning experience, (5) learning outcome, (6) 
communication skill, (7) learning place, and (8) learning time, for example: 


during end of semester. 


fe) During | go for tutor by bus, | use this time to learn 
| think it depend on the thinking at that time. When | have ro) 


both mobile and the idea, | use mobile for typing. | will use 


computer if | want to. E . : : 2 
re Either mobile or computer is similar. If typing with computer, 


it is comfortable to put the idea and correct typing. 
Nonetheless, — with mobile not easy to correct. 


? 
| would like to communicate with a foreigner but | have no 
aS fo) chance and dare. However, | would try learning this course. 


a have free time, | always learn the culture in YouTube. 


| like talking to a foreigner in order to know other people's 


Just used smartphone on the bus. Just for check. 


| always use mobile phone and the Ipad. 


? 


| almost use free time. | usually check my phone when | have 
time. 


Learning device (8) 
| usually learn at home in the night because | am a tutor \ fo) | 


Bx Y ; culture 
Ne About 10-20 minutes per time, fo 
fe) \ J a =————~T have learnt English. | think it free and not waste of time so | 
| use free time after finish my home business to learn. ——___> Ga Leanna motivation a o this course. 


——— ef? va 
| ; Learning time | think it's very comfortable because | also learn by phone while | 

| also learn while | am on the bus to go tutoring. am on the bus to go tutoring. l agree that leaming website s 

o fe) suitable for student who have not much time, 


™~ A) Just answer the discussion and overview. 


Lecturer sound in video is very clear but | have to use 


I learn at home in the night—_ a 
U-MOOG learning (0) 
ning ( qa subtitle because some words are hard to understand.¢ 


In the night | learn at home g | almost learn outside in the oe place (6) = 
Learning tool (9) | try using chat but | don’t know what to talk because we are 
N o friend already. 


At home. Normally at cd | only did that at "> yA usually ¢ at home. | watched the lecture and sometimes review it again fe) 
| always use the Line to chat about the problem with my 
When | check something, may be on the bus. friend before learning. 
When | was typing the English, | felt it wrong grammar. fo) 
o One time in China. It’s about as 
After | read his answer | feel | don’t want to say anything. q —S——— never learn the full course e-learning, just learn in English 
Everyone has loved their own country. Something | cannot—____~ @Jj Elearning Experience (4) sound lab. 
Say. aa 
oo Communication skill ( fo) 
__ 
| have worried about grammar. | usually ask my friend the I have learnt e-learning before but | never learn the 


sentence I type is ok or not. | check my typing for a long 


time. ro) 


When I type English, | don't care about grammar. | always fo) 
think how clearly to communicate. 
Before typing | have to analyze the data. | usually worry 
about the grammar, but | focus on how to a, 


clearly. o 
re 
fo) ees outcome (7) | have changed my viewpoint, but someone who didn't like 


| got the culture knowledge in video lecture and discussion before may have a few changes. 


? 


My opinion or attitude has changed since | did the second ‘\ fo) 
discussion. 


discussion style. The e-learning | learnt was talk style and 
did not type or discuss. 


| never learn an online course. fe) 


Before learning, | have view of Chinese people are not good. 
Now | open my mind for the Chinese. 


| think | use the attitude such as | change my viewpoint to 
view the Chinese do. 


Figure 2 Exploration of U-MOOC learning 
(1) Learning device 
“T think it depend on the thinking at that time. When I have both mobile 
and the idea, I use mobile for typing. I will use computer if I want to” 
Participant | 
(2) Learning motivation 
“I would like to communicate with a foreigner but I have no chance 
and dare to. However, I would try learning this course” 
Participant 5 
(3) Learning tools 
“Lecturer sound in video is very clear but I have to use subtitle because 
some words are hard to understand” 
Participant 2 
(4) E-learning experience 
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“T never learn the full course e-learning, just learn in English sound lab” 
Participant 3 
“One time in China. It is about Psychology” 
Participant 6 
(5) Learning outcome 
“Before learning, I have view of Chinese people are not good. Now I 
open my mind for the Chinese” 
Participant 3 
(6) Communication skill 
“I have worried about grammar. I usually ask my friend the sentence I 
type is ok or not. I check my typing for a long time” 
Participant 2 
(7) Learning place 
“In the night, I learn at home and I almost learn outside in the day” 
Participant | 
(8) Learning time 
“T also learn while Iam on the bus to go tutoring” 
Participant 4 


Theme 2: Cognitive dissonance, this theme was explored the dissonances that occurred while the respondent 
was learning in the MOOC, for example: 


9 


Data from website showed those are Chinese culture, | also 
think that they can talk quietly but they did not do. 


About culture, | already change my opinion such as Chinese 

talk loudly: in the past, | wonder why they do and | 

understand after Chinese explain to me. ae | have known about the manner before because some 
qi —Chinese live in my village. Then | search the internet for 


a proving data and | post to the discussion board. 


| read information on websites seriously, they inform that 
have many rules and regulation. Thus, | ask the Chinese, 
they told nowadays is not serious about the rule anymore. fe) 


Some information on internet is not update. If we talk with 
owner culture, we would really know their culture. 


Figure 3 Exploration of cognitive dissonance 


“About culture, I already change my opinion such as Chinese talk 
loudly: in the past, I wonder why they do and I understand after Chinese 
explain to me” 

Participant 5 


“T have known about the manner before because some Chinese live in 
my village. Then I search the internet for proving data and I post to the 
discussion board” 

Participant 4 
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Theme 3: Recommendations for ubiquitous MOOC, this theme was explored the recommendation about 
the U-MOOC that consist of two dimensions: (1) learning system and (2) learning strategies, for example: 


When | register on some websites, they will send you email. 
This is not. At the same time, if you have a new process, 
may be you send email to us. | have a new course and log in 
website. 


For the chat button, | think it hard to see so the learners may fe) 
ignore it. | can read subtitle on lecture video but sometimes it has a 


ORS mistake. 
a 
fo) __ Learning system (1 Dc fo) 


| think chat button should be a big pop up for easy to 
notice. | recommend the chat system should like the Facebook chat 
that have a pop up because some learners cannot see. 


Recommendations for U-MOOC (0) 
fo) | would like to see the real lecturer in classroom but online is 
| skip learning the overview because the course already has a good choice. | want both. 


the step to follow. Lee fe) 


—~G@ 
. ‘ We can have an online meeting but sometime should have 
0 ee a aay real discuss in the classroom. 


| like various media not only lecturer’s media. fo) 


Learners not diversity only have Chinese. 
Figure 4 Exploration of recommendations for U-MOOC 


(1) Learning system 
“T can read subtitle on lecture video but sometimes it has a mistake” 
Participant 2 


“T recommend the chat system should like the Facebook chat that have 
a pop up because some learners cannot see” 
Participant | 
(2) Learning strategies 
“I would like to see the real lecturer in classroom but online is a good 
choice. I want both” 
Participant 3 


“We can have an online meeting but sometime should have real discuss 
in the classroom” 
Participant 4 


Discussion 
U-MOOC Learning 


Recognizing U-MOOC learning is important. The U-MOOC learner can enhance knowledge everywhere 
every time as a result of the U-MOOC instructional model’s flexible learning. The results show that, some students 
learned in free time and the others learned at night. The findings were supported by a key-informant who stated that 
“T also learn while I am on the bus to go tutoring” and “In the night, I learn at home and I almost learn outside in the 
day” The finding was congruent with Jansen and Schuwer (2015) who stated that all features of MOOCs were 
available online, therefore participants could access the Internet anywhere. In other words, time was not a barrier to 
learn in this course. 

The results showed that the contents could be viewed on a PC, tablet, or smartphone. Some students used a 
personal computer (PC) when they were home because it was required typing for the course. Sometime they used a 
tablet or mobile phone when they were not at home. The findings were supported by a key-informant who stated that 
“Either mobile or computer is similar. If typing with computer, it is comfortable to put the idea and correct typing. 
Nonetheless, typing with mobile not easy to correct”. The finding was supportive of Mehlenbacher (2010) who stated 
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that MOOCs would require distributed technologies (e.g., laptops, mobile phones, and production handhelds) and 
wireless, connected technology (e.g., networks, Internet-enabled interaction systems). In the same way, the ubiquitous 
system in U-MOOC could support students who were uncomfortable moving to another place (McKay & Lenarcic, 
2015). 

In recognizing learning outcomes, the U-MOOC learning activities were observably and could help students 
change their cultural attitude. This outcome was supported by a key-informant who stated that “Before learning, I 
have view of Chinese people are not good. Now I open my mind for the Chinese”. The finding was congruent with 
Watson et al. (2016) who found that the MOOC used dissonance to make the attitude change. 


Cognitive dissonance 


Dissonance could be defined as a confliction between the three attitudinal components: cognitive, affective, 
and behavioral (Watson et al., 2016). The cognitive dissonance which is found in ubiquitous MOOC instructional 
design model could divided into two types. One was the dissonant that occurs during the course, and another dissonant 
was what the students already have before learning in this course. 

Cognitive dissonance concerned the strategies in U-MOOC to resolve the cultural conflict. This evidence was 
supported by two key-informants who stated that “About culture, I already change my opinion such as Chinese talk 
loudly: in the past, I wonder why they do and I understand after Chinese explain to me” and “I have known about the 
manner before because some Chinese live in my village. Then I search the internet for proving data and I post to the 
discussion board” The finding was consistent with Schunk (2012) who stated that dissonance should increase as the 
discrepancy between cognitions increases. The finding was also similar to Jonassen, Spector, Driscoll, Merrill, and 
van Merrienboer (2008) who reported that dissonant situations must be carefully selected experiences that are real to 
the learner so that the learner cannot easily dismiss the situation as untrue. The former learning dissonant occurred 
because they did not have a chance to communicate with foreign student to reduce any dissonance. Some learners 
changed their thought by new added information. Also, Cooper (2007) indicated that learners could cope with 
dissonance by changing their beliefs to eliminate the dissonance. Moreover, Schunk (2012) stated that cognitive 
dissonance can be reduced by changing a discrepant cognition. 
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